Anomaly detection is an important problem that has been researched within diverse research areas and application domains. Many anomaly detection techniques have been specifically developed for certain application domains, while others are more generic. This survey tries to provide a structured and comprehensive overview of the research on anomaly detection. We have grouped existing techniques into different categories based on the underlying approach adopted by each technique. For each category we have identified key assumptions, which are used by the techniques to differentiate between normal and anomalous behavior. When applying a given technique to a particular domain, these assumptions can be used as guidelines to assess the effectiveness of the technique in that domain. For each category, we provide a basic anomaly detection technique, and then show how the different existing techniques in that category are variants of the basic technique. This template provides an easier and more succinct understanding of the techniques belonging to each category. Further, for each category, we identify the advantages and disadvantages of the techniques in that category. We also provide a discussion on the computational complexity of the techniques since it is an important issue in real application domains. We hope that this survey will provide a better understanding of the different directions in which research has been done on this topic, and how techniques developed in one area can be applied in domains for which they were not intended to begin with.

/pdf/anomaly-detection-a-survey-1x33g9m0a3.pdf

Anomaly detection: A survey

Data Mining - Concepts and Techniques.

Matrix Factorization Techniques for Recommender Systems

We explore the effect of dimensionality on the nearest neighbor problem. We show that under a broad set of conditions (much broader than independent and identically distributed dimensions), as dimensionality increases, the distance to the nearest data point approaches the distance to the farthest data point. To provide a practical perspective, we present empirical results on both real and synthetic data sets that demonstrate that this effect can occur for as few as 10-15 dimensions. These results should not be interpreted to mean that high-dimensional indexing is never meaningful; we illustrate this point by identifying some high-dimensional workloads for which this effect does not occur. However, our results do emphasize that the methodology used almost universally in the database literature to evaluate high-dimensional indexing techniques is flawed, and should be modified. In particular, most such techniques proposed in the literature are not evaluated versus simple linear scan, and are evaluated over workloads for which nearest neighbor is not meaningful. Often, even the reported experiments, when analyzed carefully, show that linear scan would outperform the techniques being proposed on the workloads studied in high (10-15) dimensionality!.

When is nearest neighbor meaningful

Journal of Statistical Mechanics: theory and experiment

Architecture and implementation of a form document management system SPECDOQ are presented. The system automates or semi-automates manipulation of conventional form documents combining alphanumeric and Kanji/Kana character strings, texts, and graphics. SPECDOQ features uniform management of the document database in terms of unnormalized relations, named nested tables. Complicated manipulation of form documents can be accomplished by combinations of abstract, algebraic operations on the nested tables. The system also features graphics-based user friendly manipulation of external form documents on the screen. Architectural key concepts in SPECDOQ and their implementation are explained. Some system-user interaction examples and early experiences on SPECDOQ field use are also given in this paper.

Form document management system SPECDOQ its architecture and implementation

Outlier detection, a data mining technique to detect rare events, deviant objects, and exceptions from data, has drawn increasing attention in recent years. Much existing research targets record data constructed with numerical attributes or a set of points having numeric values. However, very few studies have attempted to detect outliers from data having items. We focus on transaction data and propose a framework for detecting outlier transactions that behave abnormally compared to others. As an outlier, we are interested in a transaction t in which more items are not observed even though they should normally have a strong dependency on item sets in t. We use information of association rules with high confidence for the outlier degree calculation. In this paper, we first discuss what outliers of transactions are, and provide an outlier degree for systematically detecting outlier transactions. We also propose algorithms for efficiently detecting outlier transactions from transaction databases. We present two devices for faster detection that (i) remove redundant association rules and (ii) prune candidates of outlier transactions utilizing maximal frequent itemsets. In experiments using synthetic and real world data sets, we show that our proposal can derive enough detection accuracies and detect outlier transactions faster than a brute force algorithm.

Outlier Detection for Transaction Databases using Association Rules with High Confidences

In this paper, we describe a document clustering method called novelty-based document clustering This method clusters documents based on similarity and novelty The method assigns higher weights to recent documents than old ones and generates clusters with the focus on recent topics The similarity function is derived probabilistically, extending the conventional cosine measure of the vector space model by incorporating a document forgetting model to produce novelty-based clusters The clustering procedure is a variation of the K-means method An additional feature of our clustering method is an incremental update facility, which is applied when new documents are incorporated into a document repository Performance of the clustering method is examined through experiments Experimental results show the efficiency and effectiveness of our method

https://www.db.ss.is.nagoya-u.ac.jp/~dbgroup/public_papers/2007-wwwj-khy.pdf

Hiroyuki Kitagawa

Papers

Form document management system SPECDOQ its architecture and implementation

Outlier Detection for Transaction Databases using Association Rules with High Confidences

A Novelty-based Clustering Method for On-line Documents

CAMLP: Confidence-Aware Modulated Label Propagation.

Outlier Detection for Transaction Databases Using Association Rules