Anomaly detection is an important problem that has been researched within diverse research areas and application domains. Many anomaly detection techniques have been specifically developed for certain application domains, while others are more generic. This survey tries to provide a structured and comprehensive overview of the research on anomaly detection. We have grouped existing techniques into different categories based on the underlying approach adopted by each technique. For each category we have identified key assumptions, which are used by the techniques to differentiate between normal and anomalous behavior. When applying a given technique to a particular domain, these assumptions can be used as guidelines to assess the effectiveness of the technique in that domain. For each category, we provide a basic anomaly detection technique, and then show how the different existing techniques in that category are variants of the basic technique. This template provides an easier and more succinct understanding of the techniques belonging to each category. Further, for each category, we identify the advantages and disadvantages of the techniques in that category. We also provide a discussion on the computational complexity of the techniques since it is an important issue in real application domains. We hope that this survey will provide a better understanding of the different directions in which research has been done on this topic, and how techniques developed in one area can be applied in domains for which they were not intended to begin with.

/pdf/anomaly-detection-a-survey-1x33g9m0a3.pdf

Anomaly detection: A survey

Data Mining - Concepts and Techniques.

Matrix Factorization Techniques for Recommender Systems

We explore the effect of dimensionality on the nearest neighbor problem. We show that under a broad set of conditions (much broader than independent and identically distributed dimensions), as dimensionality increases, the distance to the nearest data point approaches the distance to the farthest data point. To provide a practical perspective, we present empirical results on both real and synthetic data sets that demonstrate that this effect can occur for as few as 10-15 dimensions. These results should not be interpreted to mean that high-dimensional indexing is never meaningful; we illustrate this point by identifying some high-dimensional workloads for which this effect does not occur. However, our results do emphasize that the methodology used almost universally in the database literature to evaluate high-dimensional indexing techniques is flawed, and should be modified. In particular, most such techniques proposed in the literature are not evaluated versus simple linear scan, and are evaluated over workloads for which nearest neighbor is not meaningful. Often, even the reported experiments, when analyzed carefully, show that linear scan would outperform the techniques being proposed on the workloads studied in high (10-15) dimensionality!.

When is nearest neighbor meaningful

Journal of Statistical Mechanics: theory and experiment

Extracting New Topic Documents from Hidden Web Sites

A large quantity of DNA sequence data is being generated at high speed and in low cost by next generation sequencing (NGS) technology in recent years. NGS influences wide range of biology, including evolutionary biology, which is the field that infers the evolutionary relationships of genes or organisms from sequence data. In particular, the phylogenetic analyses using massive amount of data generated by NGS have been actively conducted. To infer the phylogenetic relationship, a number of alignments that comprise sets of sequences need to be maintained, i.e., they need to be updated whenever new sequence data become available. However, there have been no database that support updates of alignments, i.e., addition and/or removal of sequences from existing alignments. Instead, individual researchers independently update their alignments manually. To cope with this problem, we propose a system for phylogenetic analyses over alignments of NGS data. It takes as input NGS data, predicts the orthologue that each sequence belongs to, and updates the alignments. Moreover, by describing the related alignments in tree structure, it can maintain stored alignments in a systematic way. To prove the concept, we implement a prototype web application. We expect that our system help biological researchers carry out phylogenetic analysis on large scale data including those from NGS).

A System for Phylogenetic Analyses over Alignments of Next Generation Sequence Data

Due to the advance of information and communications technology and sensor technology, a large quantity of sequence data (time series data, log data, etc.) are generated and processed every day. Row pattern matching for the sequence data stored in relational databases was standardized as SQL/RPR in 2016. Today, in addition to relational databases, there are many frameworks for processing a large amount of data in parallel and distributed computing environments. They include MapReduce and Spark. Hive and Spark SQL enable us to code data analysis processes in SQL-like query languages. Row pattern matching is also beneficial in Hive and Spark SQL. However, computational cost of the row pattern matching process is large and it is needed to make this process efficient. In this paper, we propose two optimization methods to realize the reduction of computational cost for row pattern matching process. We focus on Spark and show design and implementation of the proposed methods for Spark SQL. We verify by the experiments that our optimization methods really contribute to the reduction of the processing time of Spark SQL queries including row pattern matching.

Optimization of Row Pattern Matching over Sequence Data in Spark SQL

Developments in network and sensor device technologies enable us to easily obtain real-world information, such as locations of moving objects, weather information, news, and stock prices. These data are continuously supplied, and they are regarded as data streams. Because of the dramatical increase of streaming data, their management and utilization has become more and more important. This paper describes a data stream management system named Harmonica. Harmonica employs an architecture combining our stream processing engine named stream-spinner and relational DBMSs. Based on the architecture, the system processes both continuous queries and traditional one-shot queries. Moreover, Harmonica supports continuous persistence requirements for streaming data as well as queries including selection, join, projection, and user-defined functions over data streams. Users can also specify continuous queries that integrate streaming data and persistent data stored in databases. Using the Harmonica API, users can develop a variety of applications coping with different continuous steaming data and data stored in databases. Our system can be deployed in network environments to achieve efficient and dependable distributed stream processing.

Hiroyuki Kitagawa

Papers

Extracting New Topic Documents from Hidden Web Sites

A System for Phylogenetic Analyses over Alignments of Next Generation Sequence Data

Optimization of Row Pattern Matching over Sequence Data in Spark SQL

Stream Data Management Based on Integration of a Stream Processing Engine and Databases

An Example-based Web-site Construction Tool and Its Implementation.