We explore the effect of dimensionality on the nearest neighbor problem. We show that under a broad set of conditions (much broader than independent and identically distributed dimensions), as dimensionality increases, the distance to the nearest data point approaches the distance to the farthest data point. To provide a practical perspective, we present empirical results on both real and synthetic data sets that demonstrate that this effect can occur for as few as 10-15 dimensions. These results should not be interpreted to mean that high-dimensional indexing is never meaningful; we illustrate this point by identifying some high-dimensional workloads for which this effect does not occur. However, our results do emphasize that the methodology used almost universally in the database literature to evaluate high-dimensional indexing techniques is flawed, and should be modified. In particular, most such techniques proposed in the literature are not evaluated versus simple linear scan, and are evaluated over workloads for which nearest neighbor is not meaningful. Often, even the reported experiments, when analyzed carefully, show that linear scan would outperform the techniques being proposed on the workloads studied in high (10-15) dimensionality!.

When is nearest neighbor meaningful

Search operations in databases require special support at the physical level. This is true for conventional databases as well as spatial databases, where typical search operations include the point query (find all objects that contain a given search point) and the region query (find all objects that overlap a given search region). More than ten years of spatial database research have resulted in a great variety of multidimensional access methods to support such operations. We give an overview of that work. After a brief survey of spatial data management in general, we first present the class of point access methods, which are used to search sets of points in two or more dimensions. The second part of the paper is devoted to spatial access methods to handle extended objects, such as rectangles or polyhedra. We conclude with a discussion of theoretical and experimental results concerning the relative performance of various approaches.

/pdf/multidimensional-access-methods-4jxurm5el0.pdf

Multidimensional access methods

Existing trajectory clustering algorithms group similar trajectories as a whole, thus discovering common trajectories. Our key observation is that clustering trajectories as a whole could miss common sub-trajectories. Discovering common sub-trajectories is very useful in many applications, especially if we have regions of special interest for analysis. In this paper, we propose a new partition-and-group framework for clustering trajectories, which partitions a trajectory into a set of line segments, and then, groups similar line segments together into a cluster. The primary advantage of this framework is to discover common sub-trajectories from a trajectory database. Based on this partition-and-group framework, we develop a trajectory clustering algorithm TRACLUS. Our algorithm consists of two phases: partitioning and grouping. For the first phase, we present a formal trajectory partitioning algorithm using the minimum description length(MDL) principle. For the second phase, we present a density-based line-segment clustering algorithm. Experimental results demonstrate that TRACLUS correctly discovers common sub-trajectories from real trajectory data.

/pdf/trajectory-clustering-a-partition-and-group-framework-2b61tdpjnn.pdf

Trajectory clustering: a partition-and-group framework

The last decade has witnessed a tremendous growths of interests in applications that deal with querying and mining of time series data. Numerous representation methods for dimensionality reduction and similarity measures geared towards time series have been introduced. Each individual work introducing a particular method has made specific claims and, aside from the occasional theoretical justifications, provided quantitative experimental observations. However, for the most part, the comparative aspects of these experiments were too narrowly focused on demonstrating the benefits of the proposed methods over some of the previously introduced ones. In order to provide a comprehensive validation, we conducted an extensive set of time series experiments re-implementing 8 different representation methods and 9 similarity measures and their variants, and testing their effectiveness on 38 time series data sets from a wide variety of application domains. In this paper, we give an overview of these different techniques and present our comparative experimental findings regarding their effectiveness. Our experiments have provided both a unified validation of some of the existing achievements, and in some cases, suggested that certain claims in the literature may be unduly optimistic.

/pdf/querying-and-mining-of-time-series-data-experimental-4uzriiubdj.pdf

Querying and mining of time series data: experimental comparison of representations and distance measures

A review on time series data mining

Several salient-object-based data models have been proposed to model video data. However, none of them addresses the development of an index structure to efficiently handle salient-object-based queries. There are several indexing schemes that have been proposed for spatiotemporal relationships among objects, and they are used to optimize timestamp and internal queries, which are rarely used in video databases. Moreover, these index structures are designed without consideration of the granularity levels of constraints on salient objects and the characteristics of video data. In this paper, we propose a multilevel index structure (MINDEX) to efficiently handle the salient-object-based queries with different levels of constraints. We present experimental results showing the performance of different methods of MINDEX construction.

MINDEX: An efficient index structure for salient-object-based queries in video databases

Numerous applications in different fields, such as spatial databases, multimedia databases, data mining and recommender systems, may benefit from efficient and effective aggregate similarity search, also known as aggregate nearest neighbor AggNN search. Given a group of query objects Q, the goal of AggNN is to retrieve the k most similar objects from the database, where the underlying similarity measure is defined as an aggregation usually sum, avg or max of the distances between the retrieved objects and every query object in Q. Recently, the problem was generalized so as to retrieve the k objects which are most similar to a fixed proportion of the elements of Q. This variant of aggregate similarity search is referred to as 'flexible AggNN', or FANN. In this work, we propose two approximation algorithms, one for the sum and avg variants of FANN, and the other for the max variant. Extensive experiments are provided showing that, relative to state-of-the-art approaches both exact and approximate, our algorithms produce query results with good accuracy, while at the same time being very efficient -- even for real datasets of very high dimension.

Flexible Aggregate Similarity Search in High-Dimensional Data Sets

Solar Energetic Particle events (SEPs) are among the most dangerous transient phenomena of solar activity. As hazardous radiation, SEPs may affect the health of astronauts in outer space and adversely impact current and future space exploration. In this paper, we consider the problem of daily prediction of Solar Proton Events (SPEs) based on the characteristics of the magnetic fields in solar Active Regions (ARs), preceding soft X-ray and proton fluxes, and statistics of solar radio bursts. The machine learning (ML) algorithm uses an artificial neural network of custom architecture designed for whole-Sun input. The predictions of the ML model are compared with the SWPC NOAA operational forecasts of SPEs. Our preliminary results indicate that 1) for the AR-based predictions, it is necessary to take into account ARs at the western limb and on the far side of the Sun; 2) characteristics of the preceding proton flux represent the most valuable input for prediction; 3) daily median characteristics of ARs and the counts of type II, III, and IV radio bursts may be excluded from the forecast without performance loss; and 4) ML-based forecasts outperform SWPC NOAA forecasts in situations in which missing SPE events is very undesirable. The introduced approach indicates the possibility of developing robust "all-clear" SPE forecasts by employing machine learning methods.

Prediction of Solar Proton Events with Machine Learning: Comparison with Operational Forecasts and "All-Clear" Perspectives

Using multi-scale histograms to answer pattern existence and shape match queries over time series data

In this demonstration, we show a recommender system for the Music Information Retrieval (MIR) research community. We extract the key topics and tags by analyzing the ten-year cumulative ISMIR proceedings, and recommend papers and research colleagues to users in an interactive way.

/pdf/recommender-system-for-mir-research-community-2y2x87q8sc.pdf

Vincent Oria

Papers

MINDEX: An efficient index structure for salient-object-based queries in video databases

Flexible Aggregate Similarity Search in High-Dimensional Data Sets

Prediction of Solar Proton Events with Machine Learning: Comparison with Operational Forecasts and "All-Clear" Perspectives

Using multi-scale histograms to answer pattern existence and shape match queries over time series data

Recommender system for MIR research community