Anomaly detection is an important problem that has been researched within diverse research areas and application domains. Many anomaly detection techniques have been specifically developed for certain application domains, while others are more generic. This survey tries to provide a structured and comprehensive overview of the research on anomaly detection. We have grouped existing techniques into different categories based on the underlying approach adopted by each technique. For each category we have identified key assumptions, which are used by the techniques to differentiate between normal and anomalous behavior. When applying a given technique to a particular domain, these assumptions can be used as guidelines to assess the effectiveness of the technique in that domain. For each category, we provide a basic anomaly detection technique, and then show how the different existing techniques in that category are variants of the basic technique. This template provides an easier and more succinct understanding of the techniques belonging to each category. Further, for each category, we identify the advantages and disadvantages of the techniques in that category. We also provide a discussion on the computational complexity of the techniques since it is an important issue in real application domains. We hope that this survey will provide a better understanding of the different directions in which research has been done on this topic, and how techniques developed in one area can be applied in domains for which they were not intended to begin with.

/pdf/anomaly-detection-a-survey-1x33g9m0a3.pdf

Anomaly detection: A survey

Data Mining - Concepts and Techniques.

Matrix Factorization Techniques for Recommender Systems

We explore the effect of dimensionality on the nearest neighbor problem. We show that under a broad set of conditions (much broader than independent and identically distributed dimensions), as dimensionality increases, the distance to the nearest data point approaches the distance to the farthest data point. To provide a practical perspective, we present empirical results on both real and synthetic data sets that demonstrate that this effect can occur for as few as 10-15 dimensions. These results should not be interpreted to mean that high-dimensional indexing is never meaningful; we illustrate this point by identifying some high-dimensional workloads for which this effect does not occur. However, our results do emphasize that the methodology used almost universally in the database literature to evaluate high-dimensional indexing techniques is flawed, and should be modified. In particular, most such techniques proposed in the literature are not evaluated versus simple linear scan, and are evaluated over workloads for which nearest neighbor is not meaningful. Often, even the reported experiments, when analyzed carefully, show that linear scan would outperform the techniques being proposed on the workloads studied in high (10-15) dimensionality!.

When is nearest neighbor meaningful

Journal of Statistical Mechanics: theory and experiment

Outlier detection has many applications like fraud detection, medical analysis, etc. Recently, several methods for finding outliers in large datasets have been reported. These existing techniques traditionally detect based on some prescribed definitions of outliers. However, it is very difficult for a user to decide the definition of outliers in prior. Usually, they have a few outlier examples in hand, and want to find more objects just like those examples. To solve this problem, we propose a novel method to detect outliers adaptive to users’ intensions implied by the outlier examples. This is, to the best of our knowledge, the first that detect outliers based on user-provided examples. Our experiments on both synthetic and real datasets show that the method has the ability to discover outliers that match the users’ intentions.

A Method of Detecting Outliers Matching User's intentions

In order to process a few thousands of streams in real-time, public clouds seem to be an excellent choice since it supports massively amount of computing resources. A public cloud may be managed by a third party and outside the firewall of the organization, which may make it questionable whether data are managed confidentially. This paper proposes a scheme that evaluates queries over encrypted data streams. Our scheme is based on CryptDB and its modification. We describe performance issues incurred by our scheme, and propose an approach to reduce the encryption costs and amounts of transmitted data size, and an approach to reduce memory usage by analyzing a plan tree in an SPE. The result of experiments showed that applying AP-1 reduces 90% memory size, while applying AP-2 reduces 11% memory size.

A security aware stream data processing scheme on the cloud and its efficient execution methods

In this paper, we present CVS (Compressed Vector Set), a fast and space-efficient data mining framework that efficiently handles both sparse and dense datasets. CVS holds a set of vectors in a compressed format and conducts primitive vector operations, such as p-norm and dot product, without decompression. By combining these primitive operations, CVS accelerates prominent data mining or machine learning algorithms including k-nearest neighbor algorithm, stochastic gradient descent algorithm on logistic regression, and kernel methods. In contrast to the commonly used sparse matrix/vector representation, which is not effective for dense datasets, CVS efficiently handles sparse datasets and dense datasets in a unified manner. Our experimental results demonstrate that CVS can process both dense datasets and sparse datasets faster than conventional sparse vector representation with smaller memory usage.

https://www.jstage.jst.go.jp/article/ipsjjip/26/0/26_416/_pdf

Compressed Vector Set: A Fast and Space-Efficient Data Mining Framework

Outlier detection in the high-dimensional data stream is a challenging data mining task. In high-dimensional data, the distance-based measures of outlierness become less effective and unreliable. Angle-based outlier detection ABOD technique was proposed as a more suitable scheme for high-dimensional data. However, ABOD is designed for static datasets and its naive application on a sliding window over data streams will result in poor performance. In this research, we propose two incremental algorithms for fast outlier detection based on an outlier threshold value in high-dimensional data streams: IncrementalVOA and \(VOA^{*}\). IncrementalVOA is a basic incremental algorithm for computing outlier factor of each data point in each window. \(VOA^{*}\) enhances the incremental computation by using a bound-based pruning method and a retrospect-based incremental computation technique. The effectiveness and efficiency of the proposed algorithms are experimentally evaluated on synthetic and real world datasets where \(VOA^{*}\) outperformed other methods.

VOA*: Fast Angle-Based Outlier Detection over High-Dimensional Data Streams

How can we collect non-geotagged tweets posted by users in a specific location as many as possible in a limited time span? How can we find such users if we do not have much information about the specified location? Although there are varieties of methods to estimate the locations of users, these methods are not directly applicable to this problem because they require collecting a large amount of random tweets and then filter them to obtain a small amount of tweets from such users. In this paper, we propose a framework that incrementally finds such users and continuously collects tweets from them. Our framework is based on the bandit algorithm that adjusts the trade-off between exploration and exploitation, in other words, it simultaneously finds new users in the specified location and collects tweets from already-found users. The experimental results show that the bandit algorithm works well on this problem and outperforms the carefully-designed baselines.

Hiroyuki Kitagawa

Papers

A Method of Detecting Outliers Matching User's intentions

A security aware stream data processing scheme on the cloud and its efficient execution methods

Compressed Vector Set: A Fast and Space-Efficient Data Mining Framework

VOA*: Fast Angle-Based Outlier Detection over High-Dimensional Data Streams

Collecting Non-Geotagged Local Tweets via Bandit Algorithms