Anomaly detection is an important problem that has been researched within diverse research areas and application domains. Many anomaly detection techniques have been specifically developed for certain application domains, while others are more generic. This survey tries to provide a structured and comprehensive overview of the research on anomaly detection. We have grouped existing techniques into different categories based on the underlying approach adopted by each technique. For each category we have identified key assumptions, which are used by the techniques to differentiate between normal and anomalous behavior. When applying a given technique to a particular domain, these assumptions can be used as guidelines to assess the effectiveness of the technique in that domain. For each category, we provide a basic anomaly detection technique, and then show how the different existing techniques in that category are variants of the basic technique. This template provides an easier and more succinct understanding of the techniques belonging to each category. Further, for each category, we identify the advantages and disadvantages of the techniques in that category. We also provide a discussion on the computational complexity of the techniques since it is an important issue in real application domains. We hope that this survey will provide a better understanding of the different directions in which research has been done on this topic, and how techniques developed in one area can be applied in domains for which they were not intended to begin with.

/pdf/anomaly-detection-a-survey-1x33g9m0a3.pdf

Anomaly detection: A survey

Data Mining - Concepts and Techniques.

Matrix Factorization Techniques for Recommender Systems

We explore the effect of dimensionality on the nearest neighbor problem. We show that under a broad set of conditions (much broader than independent and identically distributed dimensions), as dimensionality increases, the distance to the nearest data point approaches the distance to the farthest data point. To provide a practical perspective, we present empirical results on both real and synthetic data sets that demonstrate that this effect can occur for as few as 10-15 dimensions. These results should not be interpreted to mean that high-dimensional indexing is never meaningful; we illustrate this point by identifying some high-dimensional workloads for which this effect does not occur. However, our results do emphasize that the methodology used almost universally in the database literature to evaluate high-dimensional indexing techniques is flawed, and should be modified. In particular, most such techniques proposed in the literature are not evaluated versus simple linear scan, and are evaluated over workloads for which nearest neighbor is not meaningful. Often, even the reported experiments, when analyzed carefully, show that linear scan would outperform the techniques being proposed on the workloads studied in high (10-15) dimensionality!.

When is nearest neighbor meaningful

Journal of Statistical Mechanics: theory and experiment

This paper describes a tool for providing transparent access to online meteorological databases by way of local file system. The tool is based on FUSE, an implementation of usermode filesystem on Linux, and a user is allowed to deal with online meteorological data as if they were stored in his/her local file systems. The main target of this tool is scientists in meteorology those who are not familiar with computers and related technologies. For this reason, we attempt to provide easy ways for configuring, maintaining, and using the system. For speeding up the data transfer, we integrate a caching mechanism inside the system thereby making the number of HTTP requests smaller.

A FUSE-Based Tool for Accessing Meteorological Data in Remote Servers

The demands for graph data analysis methods, e.g., clustering and ranking, are increasing. RankClus is a framework to extract good clusters by integrating clustering and ranking on heterogeneous graphs; it enhances the clustering results by alternately updates the results of clustering and ranking for the better understanding of the clusters. However, RankClus is computationally expensive if a graph is large since it needs to iterate both clustering and ranking for all nodes. In this paper, to address this problem, we propose a novel fast RankClus algorithm for heterogeneous graphs. To speed up the entire procedure of RankClus, our proposed algorithm reduces the computational cost of the ranking process in each iteration. Our proposal measures how each node affects the clustering result; if it is not significant, we prune the node. Our extensive evaluations showed that our proposal is much faster than the original method.

Fast Algorithm for Integrating Clustering with Ranking on Heterogeneous Graphs

Applying faceted search for XML data enables users to search XML data in an interactive manner. However, applying faceted search is challenging, because faceted search requires target subtrees (objects) and facets to be defined before-hand. To this problem, existing works assume that such objects and/or facets are defined manually, but it is infeasible to manually specify objects and facets in particular when the XML data are huge and/or its structure is quite complicated. To address this problem, this paper proposes an automatic extraction scheme of objects and facets from XML data. We propose two approaches, namely frequency-based approach and semantic-based approach, and also hybrid approach of them. The basic ideas of these approaches are that the frequently occurring XML elements seem to be objects and facets, and such XML elements may have semantically meaningful name. Although the proposed approaches are rather simple, the experiments using real world XML data show that the proposed approaches can automatically extract objects and facets from the XML data.

A scheme of automated object and facet extraction for faceted search over XML data

We demonstrate CrowdSheet, a spreadsheet interface for implementing complex crowdsourcing. Despite its appeal, adoption of the spreadsheet paradigm is associated with two nontrivial problems: (1) how to design the interface, which must be a natural extension of existing spreadsheets, while guaranteeing a reasonable expressive power, and (2) how to incorporate techniques for improving data quality without sacrificing the spreadsheet's easy-to-use feature. In this demo, we show three things. First, CrowdSheet allows non IT experts to easily implement crowdsourcing applications with complex workflows. Second, CrowdSheet adopts only new two spreadsheet functions to implement a fairly wide range of real-world applications. Third, its modular architecture gives CrowdSheet a declarative feature that lets users choose alternative plans for improving data quality, while keeping the CrowdSheet description simple.

CrowdSheet: Instant Implementation and Out-of-Hand Execution of Complex Crowdsourcing

Why does Smith follow Johnson on Twitter? In most cases, the reason why users follow other users is unavailable. In this work, we answer this question by proposing TagF, which analyzes the who-follows-whom network (matrix) and the who-tags-whom network (tensor) simultaneously. Concretely, our method decomposes a coupled tensor constructed from these matrix and tensor. The experimental results on million-scale Twitter networks show that TagF uncovers different, but explainable reasons why users follow other users.

Hiroyuki Kitagawa

Papers

A FUSE-Based Tool for Accessing Meteorological Data in Remote Servers

Fast Algorithm for Integrating Clustering with Ranking on Heterogeneous Graphs

A scheme of automated object and facet extraction for faceted search over XML data

CrowdSheet: Instant Implementation and Out-of-Hand Execution of Complex Crowdsourcing

Why Do You Follow Him?: Multilinear Analysis on Twitter