Anomaly detection is an important problem that has been researched within diverse research areas and application domains. Many anomaly detection techniques have been specifically developed for certain application domains, while others are more generic. This survey tries to provide a structured and comprehensive overview of the research on anomaly detection. We have grouped existing techniques into different categories based on the underlying approach adopted by each technique. For each category we have identified key assumptions, which are used by the techniques to differentiate between normal and anomalous behavior. When applying a given technique to a particular domain, these assumptions can be used as guidelines to assess the effectiveness of the technique in that domain. For each category, we provide a basic anomaly detection technique, and then show how the different existing techniques in that category are variants of the basic technique. This template provides an easier and more succinct understanding of the techniques belonging to each category. Further, for each category, we identify the advantages and disadvantages of the techniques in that category. We also provide a discussion on the computational complexity of the techniques since it is an important issue in real application domains. We hope that this survey will provide a better understanding of the different directions in which research has been done on this topic, and how techniques developed in one area can be applied in domains for which they were not intended to begin with.

/pdf/anomaly-detection-a-survey-1x33g9m0a3.pdf

Anomaly detection: A survey

Data Mining - Concepts and Techniques.

Matrix Factorization Techniques for Recommender Systems

We explore the effect of dimensionality on the nearest neighbor problem. We show that under a broad set of conditions (much broader than independent and identically distributed dimensions), as dimensionality increases, the distance to the nearest data point approaches the distance to the farthest data point. To provide a practical perspective, we present empirical results on both real and synthetic data sets that demonstrate that this effect can occur for as few as 10-15 dimensions. These results should not be interpreted to mean that high-dimensional indexing is never meaningful; we illustrate this point by identifying some high-dimensional workloads for which this effect does not occur. However, our results do emphasize that the methodology used almost universally in the database literature to evaluate high-dimensional indexing techniques is flawed, and should be modified. In particular, most such techniques proposed in the literature are not evaluated versus simple linear scan, and are evaluated over workloads for which nearest neighbor is not meaningful. Often, even the reported experiments, when analyzed carefully, show that linear scan would outperform the techniques being proposed on the workloads studied in high (10-15) dimensionality!.

When is nearest neighbor meaningful

Journal of Statistical Mechanics: theory and experiment

Many data streams are provided through the network today, and continuous queries are often used to extract useful information from data streams. When a system must process many queries continuously, query optimization is quite important for their efficient execution. In this paper, we propose a novel multiple query optimization method for continuous queries based on query execution pattern analysis. In the advanced stream processing environment assumed in the paper, we use window operators to specify time intervals to select information of interest and the execution time specification to designate when the query should be evaluated. Queries having the same operators may share many intermediate results when they are executed at close instants, but may involve only disjoint data when executed at completely different instants. Thus, query execution timing as well as common subexpressions is a key to deciding an efficient query execution plan. The basic idea of the proposed method is to identify query execution patterns from data arrival logs of data streams and to make the most of the information in deciding an efficient query execution plan. The proposed query optimization scheme first analyzes data arrival logs and extracts query execution patterns. It then forms clusters of continuous queries such that queries in the same cluster are likely to be executed at close instants. Finally, it extracts common subexpressions from among queries in each cluster and decides the query execution plan. We also show experiment results using the prototype implementation, and discuss effectiveness of the proposed approach.

A multiple continuous query optimization method based on query execution pattern analysis

Optimization of Join-Type Queries in Nested Relational Databases

あらまし 近年,地図情報の電子化や位置計測技術の発展,携帯端末などの普及を受けて,携帯端末をもって 移動する利用者やナビゲーション機器を備えた自動車などの移動オブジェクトに対し,移動状況に即した周辺情 報の提供を行う手法の開発が求められている.一般に,移動オブジェクトに周辺情報を提供する場合,移動体の 現在位置を中心として決められた範囲を検索をすることが考えられる.しかし,これまで通ってきた経路やこれ から通る予定経路が明らかであれば,それらの情報を利用してより適切な情報が提供できる余地がある.そこで 我々の研究グループでは,移動経路や移動速度などを考慮して移動オブジェクトに対して情報を提供するための 空間データベース検索モデルの開発を行い,提案した検索モデルに基づく移動オブジェクトに対する周辺情報提 供モデルを,商用の GISソフトウェア上で実装した.本論文はこのシステムの設計について述べ,実験をもとに その有用性について評価する. キーワード 移動オブジェクト,周辺情報提供,空間データベース,空間問合せ,楕円体距離

Development and Evaluation of a Spatial Database Retrieval System to Provide Neighborhood Information to Moving Objects

Data integrity constraints are fundamental in various applications, such as data management, integration, cleaning, and schema extraction. In this paper, we address the problem of finding inclusion dependencies on the Web. The problem is important because (1) applications of inclusion dependencies, such as data quality management, are beneficial in the Web context, and (2) such dependencies are not explicitly given in general. In our approach, we enumerate pairs of HTML/XML elements that possibly represent inclusion dependencies and then rank the results for verification. First, we propose a bit-based signature scheme to efficiently select candidates (element pairs) in the enumeration process. The signature scheme is unique in that it supports Jaccard containment to deal with the incomplete nature of data on the Web, and preserves the semiorder inclusion relationship among sets of words. Second, we propose a ranking scheme to support a user in checking whether each enumerated pair actually suggests inclusion dependencies. The ranking scheme sorts the enumerated pairs so that we can examine a small number of pairs for simultaneously verifying many pairs.

/pdf/efficient-filtering-and-ranking-schemes-for-finding-2wtwnjthji.pdf

Efficient filtering and ranking schemes for finding inclusion dependencies on the web

Recently, Web mining that tries to find useful knowledge from the vast amount of Web pages has attracted a lot of research interests. Besides, it is becoming an essential task to provide Web pages related to a user-specified geographic area. In this paper, we propose an approach to extract spatial information hubs from the Web. A spatial information hub is a Web page which is related to a specified geographic area and has much local information and/or many hyperlinks to local Web pages. In the traditional approach of Web link analysis, the importance and quality of pages are judged only by their contents and hyperlink structures. However, we take their geographic localities into consideration. In our approach, we first extract geographic information from Web pages to create spatial nodes and spatial links, then conduct a link analysis based on the extended link structures. We also show our approach works well based on the experiments

Hiroyuki Kitagawa

Papers

A multiple continuous query optimization method based on query execution pattern analysis

Optimization of Join-Type Queries in Nested Relational Databases

Development and Evaluation of a Spatial Database Retrieval System to Provide Neighborhood Information to Moving Objects

Efficient filtering and ranking schemes for finding inclusion dependencies on the web

Extended Link Analysis for Extracting Spatial Information Hubs