We explore the effect of dimensionality on the nearest neighbor problem. We show that under a broad set of conditions (much broader than independent and identically distributed dimensions), as dimensionality increases, the distance to the nearest data point approaches the distance to the farthest data point. To provide a practical perspective, we present empirical results on both real and synthetic data sets that demonstrate that this effect can occur for as few as 10-15 dimensions. These results should not be interpreted to mean that high-dimensional indexing is never meaningful; we illustrate this point by identifying some high-dimensional workloads for which this effect does not occur. However, our results do emphasize that the methodology used almost universally in the database literature to evaluate high-dimensional indexing techniques is flawed, and should be modified. In particular, most such techniques proposed in the literature are not evaluated versus simple linear scan, and are evaluated over workloads for which nearest neighbor is not meaningful. Often, even the reported experiments, when analyzed carefully, show that linear scan would outperform the techniques being proposed on the workloads studied in high (10-15) dimensionality!.

When is nearest neighbor meaningful

Search operations in databases require special support at the physical level. This is true for conventional databases as well as spatial databases, where typical search operations include the point query (find all objects that contain a given search point) and the region query (find all objects that overlap a given search region). More than ten years of spatial database research have resulted in a great variety of multidimensional access methods to support such operations. We give an overview of that work. After a brief survey of spatial data management in general, we first present the class of point access methods, which are used to search sets of points in two or more dimensions. The second part of the paper is devoted to spatial access methods to handle extended objects, such as rectangles or polyhedra. We conclude with a discussion of theoretical and experimental results concerning the relative performance of various approaches.

/pdf/multidimensional-access-methods-4jxurm5el0.pdf

Multidimensional access methods

Existing trajectory clustering algorithms group similar trajectories as a whole, thus discovering common trajectories. Our key observation is that clustering trajectories as a whole could miss common sub-trajectories. Discovering common sub-trajectories is very useful in many applications, especially if we have regions of special interest for analysis. In this paper, we propose a new partition-and-group framework for clustering trajectories, which partitions a trajectory into a set of line segments, and then, groups similar line segments together into a cluster. The primary advantage of this framework is to discover common sub-trajectories from a trajectory database. Based on this partition-and-group framework, we develop a trajectory clustering algorithm TRACLUS. Our algorithm consists of two phases: partitioning and grouping. For the first phase, we present a formal trajectory partitioning algorithm using the minimum description length(MDL) principle. For the second phase, we present a density-based line-segment clustering algorithm. Experimental results demonstrate that TRACLUS correctly discovers common sub-trajectories from real trajectory data.

/pdf/trajectory-clustering-a-partition-and-group-framework-2b61tdpjnn.pdf

Trajectory clustering: a partition-and-group framework

The last decade has witnessed a tremendous growths of interests in applications that deal with querying and mining of time series data. Numerous representation methods for dimensionality reduction and similarity measures geared towards time series have been introduced. Each individual work introducing a particular method has made specific claims and, aside from the occasional theoretical justifications, provided quantitative experimental observations. However, for the most part, the comparative aspects of these experiments were too narrowly focused on demonstrating the benefits of the proposed methods over some of the previously introduced ones. In order to provide a comprehensive validation, we conducted an extensive set of time series experiments re-implementing 8 different representation methods and 9 similarity measures and their variants, and testing their effectiveness on 38 time series data sets from a wide variety of application domains. In this paper, we give an overview of these different techniques and present our comparative experimental findings regarding their effectiveness. Our experiments have provided both a unified validation of some of the existing achievements, and in some cases, suggested that certain claims in the literature may be unduly optimistic.

/pdf/querying-and-mining-of-time-series-data-experimental-4uzriiubdj.pdf

Querying and mining of time series data: experimental comparison of representations and distance measures

A review on time series data mining

In recent years, a lot of research effort has been dedicated to the management of quality of service (QoS), mainly in the fields of telecommunication networks and multimedia systems. Emerging applications such as electronic commerce, health-care applications, digital publishing or data mining also have requirements regarding the quality of service, the cost of service, the quality of data to be delivered, the accuracy and precision of the retrieved data. These examples show the need to consider the concept of QoS from a broader perspective, requiring the collaboration of all the distributed system components involved. In this paper, we propose an approach to integrate user-defined QoS requirements, together with the dynamic properties of the system components involved, into a distributed query processing environment. We then propose a query optimization strategy in which multiple goals may be considered with separate cost models. Furthermore, we discuss some experiment results confirming the effectiveness of our approach.

/pdf/pushing-quality-of-service-information-and-requirements-into-3t3yidb617.pdf

Pushing quality of service information and requirements into global query optimization

Most image database prototypes and products focus mainly on similarity searches over syntactic features of images. DISIMA aims at providing querying on both syntactic and semantic features of images. The content of an image is viewed as a set of salient objects (regions of interest). Salient objects are organized into two levels: physical salient objects that store syntactic features and logical salient objects that give the semantics. DISIMA integrates a declarative query language (MOQL) and a visual query language (VisualMOQL).

DISIMA: an object-oriented approach to developing an image database system

Neighborhood graphs are an essential component of many established methods for content-based image retrieval and automated image annotation. The performance of such methods relies heavily on the semantic quality of the graphs, which can be measured as the proportion of neighbors sharing the same class label as their query images. In this paper, we propose a new framework for the efficient construction of K-nearest neighbor (K-NN) graphs based on nearest-neighbor descent (NN-Descent), in which selective sparsification of object feature vectors is interleaved with neighborhood refinement operations in an effort to improve the semantic quality of the result. A local variant of the Laplacian Score is used to identify noisy features with respect to individual images, whose values are then set to 0 (the global mean value after standardization). We show through extensive experiments that our graph construction method is able to increase the proportion of semantically-related images over unrelated images within the neighbor sets.

Improving the Quality of K-NN Graphs for Image Databases through Vector Sparsification

Solar flares are complicated physical phenomena that are observable in a broad range of the electromagnetic spectrum, from radiowaves to $\gamma$-rays. For a more comprehensive understanding of flares, it is necessary to perform a combined multi-wavelength analysis using observations from many satellites and ground-based observatories. For efficient data search, integration of different flare lists and representation of observational data, we have developed an Interactive Multi-Instrument Database of Solar Flares (this https URL). The web accessible database is fully functional and allows the user to search for uniquely-identified flare events based on their physical descriptors and availability of observations by a particular set of instruments. Currently, the data from three primary flare lists (GOES, RHESSI and HEK) and a variety of other event catalogs (Hinode, Fermi GBM, Konus-Wind, OVSA flare catalogs, CACTus CME catalog, Filament eruption catalog) and observing logs (IRIS and Nobeyama coverage) are integrated, and an additional set of physical descriptors (temperature and emission measure) is provided along with an observing summary, data links, and multi-wavelength light curves for each flare event since January, 2002. We envision that this new tool will allow researchers to significantly speed up the search of events of interest for statistical and case studies.

Interactive Multi-Instrument Database of Solar Flares

The practicality of large-scale image indexing and querying methods depends crucially upon the availability of semantic information. The manual tagging of images with semantic information is in general very labor intensive, and existing methods for automated image annotation may not always yield accurate results. The aim of this paper is to reduce to a minimum the amount of human intervention required in the semantic annotation of images, while preserving a high degree of accuracy. Ideally, only one copy of each object of interest would be labeled manually, and the labels would then be propagated automatically to all other occurrences of the objects in the database. To this end, we propose an influence propagation strategy, SW-KProp, that requires no human intervention beyond the initial labeling of a subset of the images. SW-KProp distributes semantic information within a similarity graph defined on all images in the database: each image iteratively transmits its current label information to its neighbors, and then readjusts its own label according to the combined influences of its neighbors. SW-KProp influence propagation can be efficiently performed by means of matrix computations, provided that pairwise similarities of images are available. We also propose a variant of SW-KProp which enhances the quality of the similarity graph by selecting a reduced feature set for each prelabeled image and rebuilding its neighborhood. The performances of the SW-KProp method and its variant were evaluated against several competing methods on classification tasks for three image datasets: a handwritten digit dataset, a face dataset and a web image dataset. For the digit images, SW-KProp and its variant performed consistently better than the other methods tested. For the face and web images, SW-KProp outperformed its competitors for the case when the number of prelabeled images was relatively small. The performance was seen to improve significantly when the feature selection strategy was applied.

Vincent Oria

Papers

Pushing quality of service information and requirements into global query optimization

DISIMA: an object-oriented approach to developing an image database system

Improving the Quality of K-NN Graphs for Image Databases through Vector Sparsification

Interactive Multi-Instrument Database of Solar Flares

Annotation propagation in image databases using similarity graphs