Anomaly detection is an important problem that has been researched within diverse research areas and application domains. Many anomaly detection techniques have been specifically developed for certain application domains, while others are more generic. This survey tries to provide a structured and comprehensive overview of the research on anomaly detection. We have grouped existing techniques into different categories based on the underlying approach adopted by each technique. For each category we have identified key assumptions, which are used by the techniques to differentiate between normal and anomalous behavior. When applying a given technique to a particular domain, these assumptions can be used as guidelines to assess the effectiveness of the technique in that domain. For each category, we provide a basic anomaly detection technique, and then show how the different existing techniques in that category are variants of the basic technique. This template provides an easier and more succinct understanding of the techniques belonging to each category. Further, for each category, we identify the advantages and disadvantages of the techniques in that category. We also provide a discussion on the computational complexity of the techniques since it is an important issue in real application domains. We hope that this survey will provide a better understanding of the different directions in which research has been done on this topic, and how techniques developed in one area can be applied in domains for which they were not intended to begin with.

/pdf/anomaly-detection-a-survey-1x33g9m0a3.pdf

Anomaly detection: A survey

Data Mining - Concepts and Techniques.

Matrix Factorization Techniques for Recommender Systems

We explore the effect of dimensionality on the nearest neighbor problem. We show that under a broad set of conditions (much broader than independent and identically distributed dimensions), as dimensionality increases, the distance to the nearest data point approaches the distance to the farthest data point. To provide a practical perspective, we present empirical results on both real and synthetic data sets that demonstrate that this effect can occur for as few as 10-15 dimensions. These results should not be interpreted to mean that high-dimensional indexing is never meaningful; we illustrate this point by identifying some high-dimensional workloads for which this effect does not occur. However, our results do emphasize that the methodology used almost universally in the database literature to evaluate high-dimensional indexing techniques is flawed, and should be modified. In particular, most such techniques proposed in the literature are not evaluated versus simple linear scan, and are evaluated over workloads for which nearest neighbor is not meaningful. Often, even the reported experiments, when analyzed carefully, show that linear scan would outperform the techniques being proposed on the workloads studied in high (10-15) dimensionality!.

When is nearest neighbor meaningful

Journal of Statistical Mechanics: theory and experiment

This paper proposes a working framework and a query language to support probabilistic queries for composite event detection over probabilistic event streams. The language allows users to express Kleene closure patterns for complex event detection in the physical world. Our processing method first detects sequence patterns over probabilistic data streams using AIG, a new data structure, which handles active states with a nondeterministic finite automaton (NFA). Our method then computes the probability of each detected sequence pattern based on its lineage. Through the benefit of lineage, the probability of an output event can be directly calculated without taking into account the query plan. An optimised plan can be selected. Finally, we conducted a performance evaluation of our method and compared the results with the original and optimised query plan. The experiment clearly showed that our proposal outperforms straight-forward query plans.

Efficient probabilistic event stream processing with lineage and Kleene-plus

This paper addresses the problem of tagging users in Twitter, one of the most popular microblogs. Although there are an enormous number of Twitter users, some are particularly influential regarding certain topics (e.g., politics, sports). These users often transmit useful information about their topics. For example, a user familiar with political issues often transmits useful information about the latest political news. To obtain useful information, therefore, it is very important to know these user topics. To discover user topics, we propose a user tagging method using Twitter lists, the official Twitter function for making and sharing user lists. From our observations, users included in the same list were likely to have posted on the same topic. This topic was often described by the list name. For example, the list named 'politicians-list' has politicians as its members. For this reason, our proposed method regards list names as sequences of tags and assigns them to list members. Experiments conducted using two datasets showed that our proposed method works effectively in the user profiling domain and the user ranking domain.

Tagging users based on Twitter lists

Integration of the World Wide Web and other information sources is strongly required in the recent advanced application environments. Although information sources contain various types of data objects, the volume of multimedia objects has been increasing drastically. The paper proposes a multimedia integration scheme using SMIL in a mediator based integration system. The scheme achieves dynamic creation of various Web views combining multimedia objects stored in heterogeneous information sources such as the Web, structured documents, and databases. A key point is to utilize SMIL as a framework for dynamic multimedia integration. Since there already exist some players for SMIL documents, dynamic integration results including multimedia objects can be rendered in such players, without special-purpose ones, and are ready to be broadly distributed. The main contributions of the paper are as follows. (1) We propose a dynamic, SMIL-based multimedia integration scheme. This scheme utilizes an extended relational data model named WebNR/SD. (2) We report on implementation of the proposed scheme. We have already implemented the first prototype system which realizes integration of heterogeneous information sources including multimedia objects. Actually, it can construct SMIL-based multimedia views dynamically. This shows the practical significance of our scheme.

/pdf/dynamic-creation-of-multimedia-web-views-on-heterogeneous-4vxwrc2nmc.pdf

Dynamic creation of multimedia Web views on heterogeneous information sources

Purpose




Linked data (LD) has promoted publishing information, and links published information. There are increasing number of LD datasets containing numerical data such as statistics. For this reason, analyzing numerical facts on LD has attracted attentions from diverse domains. This paper aims to support analytical processing for LD data.




Design/methodology/approach




This paper proposes a framework called H-SPOOL which provides series of SPARQL (SPARQL Protocol and RDF Query Language) queries extracting objects and attributes from LD data sets, converts them into star/snowflake schemas and materializes relevant triples as fact and dimension tables for online analytical processing (OLAP).




Findings




The applicability of H-SPOOL is evaluated using exiting LD data sets on the Web, and H-SPOOL successfully processes the LD data sets to ETL (Extract, Transform, and Load) for OLAP. Besides, experiments show that H-SPOOL reduces the number of downloaded triples comparing with existing approach.




Originality/value




H-SPOOL is the first work for extracting OLAP-related information from SPARQL endpoints, and H-SPOOL drastically reduces the amount of downloaded triples.

H-SPOOL: A SPARQL-based ETL framework for OLAP over linked data with dimension hierarchy extraction

Keyword search over relational streams is useful when allowing users to query on streams without understanding the details about the streams and query language as well. There have been several research works on this direction, and the state-of-the-art approaches exploit Candidate Networks CNs, which are schema-level descriptions of possible joining networks of tuples, and generate query plans based on CNs. However, in fact, the performance of these approaches seriously degrades in particular when the maximum size of CNs $$T_{max}$$ and/or the number of query keywords are large due to the explosive increase in number of CNs. To cope with this problem, we propose a novel query plan called MX-structure to consolidate all CNs as much as possible. We suppress explosive blowup of nodes in query plans by consolidating all common edges among CNs. The experimental results prove that the proposed algorithm performs much better than the state-of-the-art approaches.

Hiroyuki Kitagawa

Papers

Efficient probabilistic event stream processing with lineage and Kleene-plus

Tagging users based on Twitter lists

Dynamic creation of multimedia Web views on heterogeneous information sources

H-SPOOL: A SPARQL-based ETL framework for OLAP over linked data with dimension hierarchy extraction

An Improved Method of Keyword Search over Relational Data Streams by Aggressive Candidate Network Consolidation