Anomaly detection is an important problem that has been researched within diverse research areas and application domains. Many anomaly detection techniques have been specifically developed for certain application domains, while others are more generic. This survey tries to provide a structured and comprehensive overview of the research on anomaly detection. We have grouped existing techniques into different categories based on the underlying approach adopted by each technique. For each category we have identified key assumptions, which are used by the techniques to differentiate between normal and anomalous behavior. When applying a given technique to a particular domain, these assumptions can be used as guidelines to assess the effectiveness of the technique in that domain. For each category, we provide a basic anomaly detection technique, and then show how the different existing techniques in that category are variants of the basic technique. This template provides an easier and more succinct understanding of the techniques belonging to each category. Further, for each category, we identify the advantages and disadvantages of the techniques in that category. We also provide a discussion on the computational complexity of the techniques since it is an important issue in real application domains. We hope that this survey will provide a better understanding of the different directions in which research has been done on this topic, and how techniques developed in one area can be applied in domains for which they were not intended to begin with.

/pdf/anomaly-detection-a-survey-1x33g9m0a3.pdf

Anomaly detection: A survey

Data Mining - Concepts and Techniques.

Matrix Factorization Techniques for Recommender Systems

We explore the effect of dimensionality on the nearest neighbor problem. We show that under a broad set of conditions (much broader than independent and identically distributed dimensions), as dimensionality increases, the distance to the nearest data point approaches the distance to the farthest data point. To provide a practical perspective, we present empirical results on both real and synthetic data sets that demonstrate that this effect can occur for as few as 10-15 dimensions. These results should not be interpreted to mean that high-dimensional indexing is never meaningful; we illustrate this point by identifying some high-dimensional workloads for which this effect does not occur. However, our results do emphasize that the methodology used almost universally in the database literature to evaluate high-dimensional indexing techniques is flawed, and should be modified. In particular, most such techniques proposed in the literature are not evaluated versus simple linear scan, and are evaluated over workloads for which nearest neighbor is not meaningful. Often, even the reported experiments, when analyzed carefully, show that linear scan would outperform the techniques being proposed on the workloads studied in high (10-15) dimensionality!.

When is nearest neighbor meaningful

Journal of Statistical Mechanics: theory and experiment

We demonstrate how our system can help users con-struct data-intensive Web sites from XML repositories anddatabases in a very intuitive fashion. The key idea is toshow the system some example operations with data in-stances and let it infer the operations for the whole collec-tion of data objects in the data repositories. This problem ischallenging, because we allow data in the data repositoriesto be semistructured, and inferring (generalizing) the op-erations is a non-trivial task. We already found throughexperiments that even users without any knowledge ondatabase-related concepts(such as database schemas, SQL-style queries and path expressions) can construct fairlycomplex Web-sites from a collection of XML documentsand a relational database, by following their intuitions. Wethink our system is an example of how end users can beempowered by the semistructured data technologies the re-search community has been developing, because the Web-site construction process is realized by them.The demonstration is done with an implemented systemof our AQUA (Amalgamation of QUerying and Authoring)project[5] . The project started with a recognition that thewide acceptance of the world wide web is not only due toits easy way to access data but also to the simple mecha-nism that enables end users to publish their own, personalWeb contents[3]. Some of such personal Web sites are verydata-intensive: For example, movie enthusiasts manage alot of movie information, and researchers’ Web sites havetheir publications. (They are not necessarily computer sci-ence researchers.) While they write tons of HTML docu-ments, few of them use database management systems. Weattribute it to the following two causes: (1) For end users,incorporation of databases’ contents into Web sites is difﬁ-cult: They can use commercially available authoring toolswith drag-and-drop facilities todesign their sites. However,they are typically required to write queries in SQL, whichis a completely different framework from the page author-ing. (2) The nature of live information from the real world

/pdf/enabling-end-users-to-construct-data-intensive-web-sites-luz82mm4he.pdf

Enabling End-users to Construct Data-intensive Web-sites from XML Repositories: An Example-based Approach

This paper proposes a system for social reading based on EPUB3, a standardized format for EPUB publications. The system is to collect and manage reading-related information, such as EPUB publications' metadata, bookmarks, and annotations, as well as users' behavioral information, such as the time and the location at which the users read the e-books, the timing when they turned the pages, etc. The system not only allows the users to share the reading-related information with each other, but also enables them to extract useful information out of the stored data. We present an overview of the proposed social reading system based on EPUB3, followed by the prototype implementation using Readium, a reference implementation of EPUB3 reader.

A System for Social Reading based on EPUB3

Content providers have recently started adding a variety of meta data to various video programs; these data provide primitive descriptors of the video contents. Personal digest viewing that uses the meta data is a new application in the digital broadcasting era. To build personal digests, semantic program structures must be constructed and significant scenes must be identified. Digests are currently made manually at content provider sites. This is time-consuming and increases the cost. This paper proposes a way to solve these problems with a rule-based personal digest-making scheme (PDMS) that can automatically and dynamically make personal digests from the meta data. In PDMS, depending on properties of the video program contents and viewer preferences, high-level semantic program structures can be constructed from the added primitive meta data and significant scenes can be extracted. The paper illustrates a formal PDMS model. It also presents detailed evaluation results of PDMS using the contents of a professional baseball game TV program.

A Rule-Based Scheme to Make Personal Digests from Video Program Meta Data

Analyzing vehicle recorder data of electric vehicles (EVs) reveals how the EVs are used. This paper proposes an OLAP framework to support analyzing trajectories in vehicle recorder data and applies the framework to vehicle recorder data of EVs. The framework consists of ETL (extract, transform, and load) process for trajectory data and visualization for analyzing the data. The ETL process includes hierarchy definitions for spatial and temporal dimensions, as well as aggregation functions for trajectory data. In the subsequent visualization phase, the framework displays results of OLAP operations on map interface. To ensure the applicability of the framework for real applications, we apply the framework to vehicle recorder data of micro-sized EVs (or μEVs), which are smaller EVs with one or two passengers including one driver and can drive at most 100km distance without charging on the way. The application realizes that the framework successfully enables analyses on the trajectory data for real analytic requirements.

Visual Spatial-OLAP for Vehicle Recorder Data on Micro-sized Electric Vehicles

As the popularity of SNS- and GPS-equipped mobile devices rapidly grows, numerous location-based applications have emerged. A common scenario is that a large number of users change location and interests from time to time; e.g., a user watches news, blogs, and videos while moving outside. Many online services have been developed based on continuously querying spatial–keyword objects. For instance, Twitter adjusts advertisements based on the location and the content of the message a user has just tweeted. In this paper, we investigate the case of dynamic spatial–keyword objects whose locations and keywords change over time. We study the problem of continuously tracking top-
 $$k$$
 dynamic spatial–keyword objects for a given set of queries. Answering this type of queries benefits many location-aware services such as e-commerce potential customer identification, drone delivery, and self-driving stores. We develop a solution based on a grid index. To deal with the changing locations and keywords of objects, our solution first finds the set of queries whose results are affected by the change and then updates the results of these queries. We propose a series of indexing and query processing techniques to accelerate the two procedures. We also discuss batch processing to cope with the case when multiple objects change locations and keywords in a time interval and top-
 $$k$$
 results are reported afterward. Experiments on real and synthetic datasets demonstrate the efficiency of our method and its superiority over alternative solutions.

Hiroyuki Kitagawa

Papers

Enabling End-users to Construct Data-intensive Web-sites from XML Repositories: An Example-based Approach

A System for Social Reading based on EPUB3

A Rule-Based Scheme to Make Personal Digests from Video Program Meta Data

Visual Spatial-OLAP for Vehicle Recorder Data on Micro-sized Electric Vehicles

Continuous top- k spatial–keyword search on dynamic objects