Anomaly detection is an important problem that has been researched within diverse research areas and application domains. Many anomaly detection techniques have been specifically developed for certain application domains, while others are more generic. This survey tries to provide a structured and comprehensive overview of the research on anomaly detection. We have grouped existing techniques into different categories based on the underlying approach adopted by each technique. For each category we have identified key assumptions, which are used by the techniques to differentiate between normal and anomalous behavior. When applying a given technique to a particular domain, these assumptions can be used as guidelines to assess the effectiveness of the technique in that domain. For each category, we provide a basic anomaly detection technique, and then show how the different existing techniques in that category are variants of the basic technique. This template provides an easier and more succinct understanding of the techniques belonging to each category. Further, for each category, we identify the advantages and disadvantages of the techniques in that category. We also provide a discussion on the computational complexity of the techniques since it is an important issue in real application domains. We hope that this survey will provide a better understanding of the different directions in which research has been done on this topic, and how techniques developed in one area can be applied in domains for which they were not intended to begin with.

/pdf/anomaly-detection-a-survey-1x33g9m0a3.pdf

Anomaly detection: A survey

Data Mining - Concepts and Techniques.

Matrix Factorization Techniques for Recommender Systems

We explore the effect of dimensionality on the nearest neighbor problem. We show that under a broad set of conditions (much broader than independent and identically distributed dimensions), as dimensionality increases, the distance to the nearest data point approaches the distance to the farthest data point. To provide a practical perspective, we present empirical results on both real and synthetic data sets that demonstrate that this effect can occur for as few as 10-15 dimensions. These results should not be interpreted to mean that high-dimensional indexing is never meaningful; we illustrate this point by identifying some high-dimensional workloads for which this effect does not occur. However, our results do emphasize that the methodology used almost universally in the database literature to evaluate high-dimensional indexing techniques is flawed, and should be modified. In particular, most such techniques proposed in the literature are not evaluated versus simple linear scan, and are evaluated over workloads for which nearest neighbor is not meaningful. Often, even the reported experiments, when analyzed carefully, show that linear scan would outperform the techniques being proposed on the workloads studied in high (10-15) dimensionality!.

When is nearest neighbor meaningful

Journal of Statistical Mechanics: theory and experiment

The number of real-time information sources, or so-called streams, has rapidly increased, leading to a greater demand for complex analyses over streams. Although many stream analysis methods exist, aggregation is fundamental to ascertain higher levels of knowledge from raw data. In particular, sliding-window aggregation, where aggregations over sliding windows are repeatedly computed, is useful in many real-life applications. Two stacks is the state-of-the-art method to compute sliding-window aggregations incrementally with a O(1) time complexity. However, its performance seriously degrades as the window size increases due to the high overhead to maintain its index. To address this problem, this paper proposes a linear bidirectional index (L-BiX) that exploits two kinds of partial aggregations. Specifically, forward (old-to-new) and backward (new-to-old) aggregations allow efficient computations in an incremental manner. The proposed algorithm requires the same time complexity as two stacks (O(1)). Our experimental evaluation shows that the throughput of L-BiX can be faster by up to 1.71 times than that of two stacks with a 50% smaller memory footprint.

L-BiX: incremental sliding-window aggregation over data streams using linear bidirectional aggregating indexes

Comprehensibility of source code is strongly affected by identifier names, therefore software developers need to give good (e.g. meaningful but short) names to identifiers. On the other hand, giving a good name is sometimes a difficult and time-consuming task even for experienced developers. To support naming identifiers, several techniques for recommending identifier name candidates have been proposed. These techniques, however, still have challenges on the goodness of suggested candidates and limitations on applicable situations. This paper proposes a new approach to recommending method names by applying graph embedding techniques to the method call graph. The evaluation experiment confirms that the proposed technique can suggest more appropriate method name candidates in difficult situations than the state of the art approach.

Mercem: Method Name Recommendation Based on Call Graph Embedding

How can we predict Smith’s main hobby if we know the main hobby of Smith’s friends? Can we measure the confidence in our prediction if we are given the main hobby of only a few of Smith’s friends? In this paper, we focus on how to estimate the confidence on the node classification problem. Providing a confidence level for the classification problem is important because most nodes in real world networks tend to have few neighbors, and thus, a small amount of evidence. Our contributions are three-fold: (a) novel algorithm; we propose a semi-supervised learning algorithm that converges fast, and provides the confidence estimate (b) theoretical analysis; we show the solid theoretical foundation of our algorithm and the connections to label propagation and Bayesian inference (c) empirical analysis; we perform extensive experiments on three different real networks. Specifically, the experimental results demonstrate that our algorithm outperforms other algorithms on graphs with less smoothness and low label density.

SocNL: Bayesian Label Propagation with Confidence

Rapid development and ongoing research activities on mobile devices, digital cartography, and global positioning systems (GPSs) have brought us a new type of software service - location-based services for moving objects (such as people with mobile devices and vehicles with car navigation systems). Realization of location-based services requires new technologies to provide appropriate neighborhood information to moving objects. A general approach to providing neighborhood information to moving objects is to retrieve objects in the neighborhood of a moving object with a spatial query that uses the traditional Euclidean distance. However, if we know the destination and the estimated route of a moving object, we would be able to provide more appropriate information to the object. Based on this idea, we have developed adaptive spatial query generation models that take the trajectory of a moving object into consideration to retrieve desired information. In this paper, we describe the design and implementation of the neighborhood information retrieval system based on the models and evaluate its effectiveness with experiments.

Implementation and evaluation of an adaptive neighborhood information retrieval system for mobile users

Rapid development of information technology such as mobile terminals and GPS systems enabled information services that provide location-oriented information based on users' positions. In this paper, we propose an approach for the modeling of information integration applications that incorporate spatial information sources in addition to conventional information sources to provide appropriate location-oriented information to users. First, we present our approach to the modeling of spatial information sources based on the source description framework. It provides a way to represent the content and the query capability of a spatial information source in a descriptive manner. Then we show a query processing scheme that finds a combination of information sources to respond to given queries and to evaluate them efficiently.

Hiroyuki Kitagawa

Papers

L-BiX: incremental sliding-window aggregation over data streams using linear bidirectional aggregating indexes

Mercem: Method Name Recommendation Based on Call Graph Embedding

SocNL: Bayesian Label Propagation with Confidence

Implementation and evaluation of an adaptive neighborhood information retrieval system for mobile users

Source Description-Based Approach for the Modeling of Spatial Information Integration