We explore the effect of dimensionality on the nearest neighbor problem. We show that under a broad set of conditions (much broader than independent and identically distributed dimensions), as dimensionality increases, the distance to the nearest data point approaches the distance to the farthest data point. To provide a practical perspective, we present empirical results on both real and synthetic data sets that demonstrate that this effect can occur for as few as 10-15 dimensions. These results should not be interpreted to mean that high-dimensional indexing is never meaningful; we illustrate this point by identifying some high-dimensional workloads for which this effect does not occur. However, our results do emphasize that the methodology used almost universally in the database literature to evaluate high-dimensional indexing techniques is flawed, and should be modified. In particular, most such techniques proposed in the literature are not evaluated versus simple linear scan, and are evaluated over workloads for which nearest neighbor is not meaningful. Often, even the reported experiments, when analyzed carefully, show that linear scan would outperform the techniques being proposed on the workloads studied in high (10-15) dimensionality!.

When is nearest neighbor meaningful

Search operations in databases require special support at the physical level. This is true for conventional databases as well as spatial databases, where typical search operations include the point query (find all objects that contain a given search point) and the region query (find all objects that overlap a given search region). More than ten years of spatial database research have resulted in a great variety of multidimensional access methods to support such operations. We give an overview of that work. After a brief survey of spatial data management in general, we first present the class of point access methods, which are used to search sets of points in two or more dimensions. The second part of the paper is devoted to spatial access methods to handle extended objects, such as rectangles or polyhedra. We conclude with a discussion of theoretical and experimental results concerning the relative performance of various approaches.

/pdf/multidimensional-access-methods-4jxurm5el0.pdf

Multidimensional access methods

Existing trajectory clustering algorithms group similar trajectories as a whole, thus discovering common trajectories. Our key observation is that clustering trajectories as a whole could miss common sub-trajectories. Discovering common sub-trajectories is very useful in many applications, especially if we have regions of special interest for analysis. In this paper, we propose a new partition-and-group framework for clustering trajectories, which partitions a trajectory into a set of line segments, and then, groups similar line segments together into a cluster. The primary advantage of this framework is to discover common sub-trajectories from a trajectory database. Based on this partition-and-group framework, we develop a trajectory clustering algorithm TRACLUS. Our algorithm consists of two phases: partitioning and grouping. For the first phase, we present a formal trajectory partitioning algorithm using the minimum description length(MDL) principle. For the second phase, we present a density-based line-segment clustering algorithm. Experimental results demonstrate that TRACLUS correctly discovers common sub-trajectories from real trajectory data.

/pdf/trajectory-clustering-a-partition-and-group-framework-2b61tdpjnn.pdf

Trajectory clustering: a partition-and-group framework

The last decade has witnessed a tremendous growths of interests in applications that deal with querying and mining of time series data. Numerous representation methods for dimensionality reduction and similarity measures geared towards time series have been introduced. Each individual work introducing a particular method has made specific claims and, aside from the occasional theoretical justifications, provided quantitative experimental observations. However, for the most part, the comparative aspects of these experiments were too narrowly focused on demonstrating the benefits of the proposed methods over some of the previously introduced ones. In order to provide a comprehensive validation, we conducted an extensive set of time series experiments re-implementing 8 different representation methods and 9 similarity measures and their variants, and testing their effectiveness on 38 time series data sets from a wide variety of application domains. In this paper, we give an overview of these different techniques and present our comparative experimental findings regarding their effectiveness. Our experiments have provided both a unified validation of some of the existing achievements, and in some cases, suggested that certain claims in the literature may be unduly optimistic.

/pdf/querying-and-mining-of-time-series-data-experimental-4uzriiubdj.pdf

Querying and mining of time series data: experimental comparison of representations and distance measures

A review on time series data mining

In content based retrieval, relevance feedback RF is a noticeable method for reducing the "semantic gap" between the low-level features describing the content and the usually higher-level meaning of user's target. While recent RF methods based on active learning are able to identify complex target classes after relatively few iterations, they can be quite slow on very large databases. To address this scalability issue for active RF, we put forward a method that consists in the construction of an M-tree in the feature space associated to a kernet function and in performing approximate kNN hyperplane queries with the feature space M-tree. The experiments performed on two image databases show that a significant speedup can be achieved, at the expense of a limited increase in the number of feedback rounds.

/pdf/hyperplane-queries-in-a-feature-space-m-tree-for-speeding-up-2c3u7gi7ts.pdf

Hyperplane queries in a feature-space M-tree for speeding up active learning

We develop a content-based audio COver Song IdeNtification (COSIN) system to detect/group cover songs. The COSIN takes music audio content as input and performs similarity searching to locate variants of the input (i.e., cover versions). Identified cover songs are returned in the rank order according to their similarity to the input. The COSIN also incorporates a set of tools to evaluate retrieval performance so researchers can explore different retrieval schemes and parameters (e.g. recall, precision). The COSIN utilizes a suite of techniques to detect cover songs including: Pitch + Dynamic Programming (DP), Chroma + DP, and Semantic Feature Summarization (SFS) + Hash-Based Approximate Matching (HBAM). Demonstration system shows that COSIN is a very potential music content retrieval tool. Running some music retrieval schemes on COSIN platform, recent experiments with SFS + LSH Variants demonstrate a nicely balanced efficiency (search speed) v. performance (search accuracy) tradeoff.

COSIN: content-based retrieval system for cover songs

A view mechanism can help handle the complex semantics in emerging application areas such as image databases. This paper presents the view mechanism we defined for the DISIMA image database system. Since DISIMA is being developed on top of an object-oriented database system, we first propose a powerful object-oriented view mechanism based on the separation between types (interface functions) and classes that manage objects of the same type. The image view mechanism uses our object-oriented view mechanism to allow us to give different semantics to the same image. The solution is based on the distinction between physical salient objects which are interesting objects in an image and logical salient objects which are the meanings of these objects.

/pdf/defining-views-in-an-image-database-system-2gnuxwalqq.pdf

Defining Views in an Image Database System

Digital library systems and other analytic or computational applications create documents and display screens in response to user queries “dynamically” or in “real time.” These “virtual documents” do not exist in advance, and thus hypermedia features (links, comments, and bookmark anchors) must be generated “just in time”—automatically and dynamically. In addition, accessing the hypermedia features may cause target documents to be generated or re-generated. This article describes the specific challenges for virtual documents and dynamic hypermedia functionality: dynamic regeneration, and dynamic anchor re-identification and re-location. It presents Just-in-time Hypermedia Engine to support just-in-time hypermedia across digital library and other third-party applications with dynamic content, and discusses issues prompted by this research.

/pdf/supplementing-virtual-documents-with-just-in-time-hypermedia-4v4lmkxugo.pdf

Supplementing virtual documents with just-in-time hypermedia functionality

In this study we extract the deep features and investigate the compression of the Mg II k spectral line profiles observed in quiet Sun regions by NASA's IRIS satellite. The data set of line profiles used for the analysis was obtained on April 20th, 2020, at the center of the solar disc, and contains almost 300,000 individual Mg II k line profiles after data cleaning. The data are separated into train and test subsets. The train subset was used to train the autoencoder of the varying embedding layer size. The early stopping criterion was implemented on the test subset to prevent the model from overfitting. Our results indicate that it is possible to compress the spectral line profiles more than 27 times (which corresponds to the reduction of the data dimensionality from 110 to 4) while having a 4 DN (Data Number) average reconstruction error, which is comparable to the variations in the line continuum. The mean squared error and the reconstruction error of even statistical moments sharply decrease when the dimensionality of the embedding layer increases from 1 to 4 and almost stop decreasing for higher numbers. The observed occasional improvements in training for values higher than 4 indicate that a better compact embedding may potentially be obtained if other training strategies and longer training times are used. The features learned for the critical four-dimensional case can be interpreted. In particular, three of these four features mainly control the line width, line asymmetry, and line dip formation respectively. The presented results are the first attempt to obtain a compact embedding for spectroscopic line profiles and confirm the value of this approach, in particular for feature extraction, data compression, and denoising.

Vincent Oria

Papers

Hyperplane queries in a feature-space M-tree for speeding up active learning

COSIN: content-based retrieval system for cover songs

Defining Views in an Image Database System

Supplementing virtual documents with just-in-time hypermedia functionality

Compression of Solar Spectroscopic Observations: a Case Study of Mg II k Spectral Line Profiles Observed by NASA's IRIS Satellite