Anomaly detection is an important problem that has been researched within diverse research areas and application domains. Many anomaly detection techniques have been specifically developed for certain application domains, while others are more generic. This survey tries to provide a structured and comprehensive overview of the research on anomaly detection. We have grouped existing techniques into different categories based on the underlying approach adopted by each technique. For each category we have identified key assumptions, which are used by the techniques to differentiate between normal and anomalous behavior. When applying a given technique to a particular domain, these assumptions can be used as guidelines to assess the effectiveness of the technique in that domain. For each category, we provide a basic anomaly detection technique, and then show how the different existing techniques in that category are variants of the basic technique. This template provides an easier and more succinct understanding of the techniques belonging to each category. Further, for each category, we identify the advantages and disadvantages of the techniques in that category. We also provide a discussion on the computational complexity of the techniques since it is an important issue in real application domains. We hope that this survey will provide a better understanding of the different directions in which research has been done on this topic, and how techniques developed in one area can be applied in domains for which they were not intended to begin with.

/pdf/anomaly-detection-a-survey-1x33g9m0a3.pdf

Anomaly detection: A survey

Data Mining - Concepts and Techniques.

Matrix Factorization Techniques for Recommender Systems

We explore the effect of dimensionality on the nearest neighbor problem. We show that under a broad set of conditions (much broader than independent and identically distributed dimensions), as dimensionality increases, the distance to the nearest data point approaches the distance to the farthest data point. To provide a practical perspective, we present empirical results on both real and synthetic data sets that demonstrate that this effect can occur for as few as 10-15 dimensions. These results should not be interpreted to mean that high-dimensional indexing is never meaningful; we illustrate this point by identifying some high-dimensional workloads for which this effect does not occur. However, our results do emphasize that the methodology used almost universally in the database literature to evaluate high-dimensional indexing techniques is flawed, and should be modified. In particular, most such techniques proposed in the literature are not evaluated versus simple linear scan, and are evaluated over workloads for which nearest neighbor is not meaningful. Often, even the reported experiments, when analyzed carefully, show that linear scan would outperform the techniques being proposed on the workloads studied in high (10-15) dimensionality!.

When is nearest neighbor meaningful

Journal of Statistical Mechanics: theory and experiment

Outlier detection is an important problem with applications in many fields. Such applications generally process high dimensional datasets. Among the existing methods of detecting outliers, Distance-Based outlier (DB-Outlier) detection is one of the most commonly used and simplest approaches, since it detects outliers only by calculating distances between data points. However, in high dimensional space, data is sparse, so every data point becomes a good outlier candidate. A Subspace-Based method has been proposed to deal with the curse of dimensions. It shows that meaningful outliers are likely to be identified by examining the behavior of data in low dimensional projections. On the other hand, most existing methods detect outliers with parameters being determined by users in advance. Such parameters usually contain hidden user view of outliers. Example-Based outlier detection methods are presented to be promising in discovering the hidden user view of outliers. In this paper, we discuss a new technique to detect DB-Outliers in high dimensional datasets based on user examples. Our proposed method makes use of Subspace-Based and Example-Based methods to discover a subspace where user examples are outstanding more significantly than in any other subspaces, and reports DB-Outliers detected in this subspace.

DB-Outlier Detection by Example in High Dimensional Datasets

PageChaser is a system that monitors links between Web pages and searches for the new locations of moved Web pages when it finds broken links. The problem of searching for moved pages is different from typical information retrieval problems. First, it is impossible to identify the final destination until the page is actually moved, so the index-server approach is not necessarily effective. Secondly, there is a large bias about where the new address is likely to be and crawler-based solutions can be effectively implemented, avoiding the need to search the entire Web. PageChaser incorporates a comprehensive set of heuristics, some of which are novel, in a single unified framework. This paper explains the underlying ideas behind the design and development of PageChaser.

PageChaser: A Tool for the Automatic Correction of Broken Web Links

With the recent progress of spatial information technologies and communication technologies, it has become easier to track positions of a large number of moving objects in real-time. Mobility statistics plays an important role in the interactive analysis of a large collection of moving objects trajectories and its use of movement pattern prediction. The development of an effective mobility statistics measure and its efficient computation method are critical issues. Thus, we propose an approach for constructing a mobility histogram to summarize a number of moving object trajectories. The histogram is based on a mobility statistics model called the Markov chain model. To facilitate an interactive analysis performed by a user, we provide a mobility histogram data cube-like logical representation and support an OLAP-style analysis. Since trajectory data is often received continuously as a trajectory stream, we have to support dynamic histogram construction and maintenance. We introduce a tree structure as the physical representation of a histogram and present histogram construction and maintenance methods that work efficiently within the given upper-bound size. We evaluate the performance and the precision of the proposed method by means of experiments

/pdf/a-dynamic-mobility-histogram-construction-method-based-on-3m4aoc3msp.pdf

A Dynamic Mobility Histogram Construction Method Based on Markov Chains

Design and Implementation of a Form Management System APAD Using ADABAS/INQ DBMS

Control of versions and time-varying configurations of design data is a very important issue in computer-aided design environments. A new data modeling scheme to facilitate their management is proposed. The basic idea is to manage complicatedly related time-varying design data in terms of well-defined versions and configurations of conceptual design artifacts. The modeling scheme views the design database as a collection of conceptual objects and representation objects. Representation objects are abstractions of conventional design data files and configurations of conceptual objects are cast on them as the management structure. To facilitate design data management through the conceptual objects, the scheme provides constructs for modeling version variant and invariant configurations and for modular management of large configurations. Operations for design data manipulation and a mechanism for maintaining design database consistency are also presented. >

Hiroyuki Kitagawa

Papers

DB-Outlier Detection by Example in High Dimensional Datasets

PageChaser: A Tool for the Automatic Correction of Broken Web Links

A Dynamic Mobility Histogram Construction Method Based on Markov Chains

Design and Implementation of a Form Management System APAD Using ADABAS/INQ DBMS

Design data modeling with versioned conceptual configuration