We explore the effect of dimensionality on the nearest neighbor problem. We show that under a broad set of conditions (much broader than independent and identically distributed dimensions), as dimensionality increases, the distance to the nearest data point approaches the distance to the farthest data point. To provide a practical perspective, we present empirical results on both real and synthetic data sets that demonstrate that this effect can occur for as few as 10-15 dimensions. These results should not be interpreted to mean that high-dimensional indexing is never meaningful; we illustrate this point by identifying some high-dimensional workloads for which this effect does not occur. However, our results do emphasize that the methodology used almost universally in the database literature to evaluate high-dimensional indexing techniques is flawed, and should be modified. In particular, most such techniques proposed in the literature are not evaluated versus simple linear scan, and are evaluated over workloads for which nearest neighbor is not meaningful. Often, even the reported experiments, when analyzed carefully, show that linear scan would outperform the techniques being proposed on the workloads studied in high (10-15) dimensionality!.

When is nearest neighbor meaningful

We present the design, implementation, and evaluation of ArrayStore, a new storage manager for complex, parallel array processing. ArrayStore builds on prior work in the area of multidimensional data storage, but considers the new problem of supporting a parallel and more varied workload comprising not only range-queries, but also binary operations such as joins and complex user-defined functions.This paper makes two key contributions. First, it examines several existing single-site storage management strategies and array partitioning strategies to identify which combination is best suited for the array-processing workload above. Second, it develops a new and efficient storage-management mechanism that enables parallel processing of operations that must access data from adjacent partitions.We evaluate ArrayStore on over 80GB of real data from two scientific domains and real operators used in these domains. We show that ArrayStore outperforms previously proposed storage management strategies in the context of its diverse target workload.

/pdf/arraystore-a-storage-manager-for-complex-parallel-array-39cv4i7k5v.pdf

ArrayStore: a storage manager for complex parallel array processing

The MPEG-7 Multimedia Database System (MPEG-7 MMDB)

A new implementation scheme for relational tables in multidimensional databases is proposed and evaluated. The scheme implements a relational table by employing a multidimensional array. Using multidimensional arrays provides many advantages, however suffers from some problems. In our scheme, these problems are solved by an efficient scheme of record encoding based on the notion of extendible array. Our scheme exhibits good performance in space and time costs compared with conventional implementation.

An extendible array based implementation of relational tables for multi dimensional databases

In high-energy physics experiments, large particle accelerators produce enormous quantities of data, measured in hundreds of terabytes or petabytes per year, which are deposited onto tertiary storage. The experiments are designed to study the collisions of fundamental particles, called "events", each of which is represented as a point in a multi-dimensional universe. In these environments, the best retrieval performance can be achieved only if the data is clustered on the tertiary storage by all searchable attributes of the events. Since the number of these attributes is high, the underlying data-management facility must be able to cope with extremely large volumes and very high dimensionalities of data at the same time. The proposed indexing technique is designed to facilitate both clustering and efficient retrieval of high-dimensional data on tertiary storage. The structure uses an original space-partitioning scheme, which has numerous advantages over other space-partitioning techniques. While the main objective of the design is to support high-energy physics experiments, the proposed solution is appropriate for many other scientific applications.

The design of a retrieval technique for high-dimensional data on tertiary storage

This study introduces a class of region preserving space transformation (RPST) schemes for
accessing high-dimensional data. The access methods in this class differ with respect to their space-partitioning
strategies. The study develops two new static partitioning schemes that can split each
dimension of the space within linear space complexity. They also support an effective mechanism for
handling skewed data in heavily sparse spaces. The techniques are experimentally compared to the
Pyramid Technique, which is another example of static partitioning designed for high-dimensional
data. On real high-dimensional data, the proposed RPST schemes outperform the Pyramid Technique
by a significant margin.

/pdf/a-class-of-region-preserving-space-transformations-for-32lxain3ti.pdf

Jack Lukaszuk

Papers

The design of a retrieval technique for high-dimensional data on tertiary storage

A Class of Region-preserving Space Transformations for Indexing High-dimensional Data

On accessing data in high-dimensional spaces: a comparative study of three space partitioning strategies