We investigate techniques for analysis and retrieval of object trajectories in two or three dimensional space. Such data usually contain a large amount of noise, that has made previously used metrics fail. Therefore, we formalize non-metric similarity functions based on the longest common subsequence (LCSS), which are very robust to noise and furthermore provide an intuitive notion of similarity between trajectories by giving more weight to similar portions of the sequences. Stretching of sequences in time is allowed, as well as global translation of the sequences in space. Efficient approximate algorithms that compute these similarity measures are also provided. We compare these new methods to the widely used Euclidean and time warping distance functions (for real and synthetic data) and show the superiority of our approach, especially in the strong presence of noise. We prove a weaker version of the triangle inequality and employ it in an indexing structure to answer nearest neighbor queries. Finally, we present experimental results that validate the accuracy and efficiency of our approach.

/pdf/discovering-similar-multidimensional-trajectories-3ofj6nb9yv.pdf

Discovering similar multidimensional trajectories

Computational geometry

The coming years will witness dramatic advances in wireless communications as well as positioning technologies. As a result, tracking the changing positions of objects capable of continuous movement is becoming increasingly feasible and necessary. The present paper proposes a novel, R*-tree based indexing technique that supports the efficient querying of the current and projected future positions of such moving objects. The technique is capable of indexing objects moving in one-, two-, and three-dimensional space. Update algorithms enable the index to accommodate a dynamic data set, where objects may appear and disappear, and where changes occur in the anticipated positions of existing objects. A comprehensive performance study is reported.

/pdf/indexing-the-positions-of-continuously-moving-objects-3eaflrmi98.pdf

Indexing the positions of continuously moving objects

Data sets in large applications are often too massive to fit completely inside the computers internal memory. The resulting input/output communication (or I/O) between fast internal memory and slower external memory (such as disks) can be a major performance bottleneck. In this article we survey the state of the art in the design and analysis of external memory (or EM) algorithms and data structures, where the goal is to exploit locality in order to reduce the I/O costs. We consider a variety of EM paradigms for solving batched and online problems efficiently in external memory. For the batched problem of sorting and related problems such as permuting and fast Fourier transform, the key paradigms include distribution and merging. The paradigm of disk striping offers an elegant way to use multiple disks in parallel. For sorting, however, disk striping can be nonoptimal with respect to I/O, so to gain further improvements we discuss distribution and merging techniques for using the disks independently. We also consider useful techniques for batched EM problems involving matrices (such as matrix multiplication and transposition), geometric data (such as finding intersections and constructing convex hulls), and graphs (such as list ranking, connected components, topological sorting, and shortest paths). In the online domain, canonical EM applications include dictionary lookup and range searching. The two important classes of indexed data structures are based upon extendible hashing and B-trees. The paradigms of filtering and bootstrapping provide a convenient means in online data structures to make effective use of the data accessed from disk. We also reexamine some of the above EM problems in slightly different settings, such as when the data items are moving, when the data items are variable-length (e.g., text strings), or when the allocated amount of internal memory can change dynamically. Programming tools and environments are available for simplifying the EM programming task. During the course of the survey, we report on some experiments in the domain of spatial databases using the TPIE system (transparent parallel I/O programming environment). The newly developed EM algorithms and data structures that incorporate the paradigms we discuss are significantly faster than methods currently used in practice.

https://users.cs.duke.edu/~reif/courses/alglectures/vitter.papers/Vit.IO_survey.pdf

External memory algorithms and data structures: dealing with massive data

High-dimensional data in Euclidean space pose special challenges to data mining algorithms. These challenges are often indiscriminately subsumed under the term ‘curse of dimensionality’, more concrete aspects being the so-called ‘distance concentration effect’, the presence of irrelevant attributes concealing relevant information, or simply efficiency issues. In about just the last few years, the task of unsupervised outlier detection has found new specialized solutions for tackling high-dimensional data in Euclidean space. These approaches fall under mainly two categories, namely considering or not considering subspaces (subsets of attributes) for the definition of outliers. The former are specifically addressing the presence of irrelevant attributes, the latter do consider the presence of irrelevant attributes implicitly at best but are more concerned with general issues of efficiency and effectiveness. Nevertheless, both types of specialized outlier detection algorithms tackle challenges specific to high-dimensional data. In this survey article, we discuss some important aspects of the ‘curse of dimensionality’ in detail and survey specialized algorithms for outlier detection from both categories. © 2012 Wiley Periodicals, Inc. Statistical Analysis and Data Mining, 2012 © 2012 Wiley Periodicals, Inc.

A survey on unsupervised outlier detection in high-dimensional numerical data

Presents the results from an extensive comparison study of three R-tree packing algorithms: the Hilbert and nearest-X packing algorithms, and an algorithm which is very simple to implement, called the STR (Sort-Tile-Recursive) algorithm The algorithms are evaluated using both synthetic and actual data from various application domains including VLSI design, GIS (Tiger files), and computational fluid dynamics Our studies also consider the impact that various degrees of buffering have on query performance Experimental results indicate that none of the algorithms as best for all types of data In general, our new algorithm requires up to 50% fewer disk accesses than the best previously proposed algorithm for point and region queries on uniformly distributed or mildly skewed point and region data, and approximately the same for highly skewed point and region data

STR: a simple and efficient algorithm for R-tree packing

CM-line loading of R-trees is useful to improve node utilization and query performance. WTepresent an algorithm for bulk loading R-trees -r&id diifers horn previous ones in two aspects (a) it partitions input data into subtrees in a top-down fashion (based on the fact that splits close to the root are likely to have a greater impact on performance), (b) at each tree level, it considers all cuts orthogonal to the coordinate axes that result in packed trees and greedily picks those optimizing an arbitrary cost function. EMm.sive esperirnentation with both real and synthetic data indicate that for region data our algorithm requires up to three times fewer disk accesses than other algorithms. It is the method of choice for data with skew in locations, areas, or aspect ratios. Such data is common in practice. Let n = number of input rectangles Let S = maximumnumber of rectangles per subtree Let M = maximumnumber of entries per node Let f (rl, r2) be the “user-supplied” cost function If n < S return {stop condition} For each dimension d For each ordering considered in this dimension d For i from 1 to [n/iVfl – 1 Let B. = MSR of first i S rectangles Let B1 = MSR of the other rectangles Remember i if f(Bo, BI) is better valued Split input set and orderings at best position.

A greedy algorithm for bulk loading R-trees

We present a new approach for approximate nearest neighbor queries for sets of high dimensional points under any L/sub t/-metric, t=1,...,/spl infin/. The proposed algorithm is efficient and simple to implement. The algorithm uses multiple shifted copies of the data points and stores them in up to (d+1) B-trees where d is the dimensionality of the data, sorted according to their position along a space filling curve. This is done in a way that allows us to guarantee that a neighbor within an O(d/sup 1+1/t/) factor of the exact nearest, can be returned with at most (d+1)log, n page accesses, where p is the branching factor of the B-trees. In practice, for real data sets, our approximate technique finds the exact nearest neighbor between 87% and 99% of the time and a point no farther than the third nearest neighbor between 98% and 100% of the time. Our solution is dynamic, allowing insertion or deletion of points in O(d log/sub p/ n) page accesses and generalizes easily to find approximate k-nearest neighbors.

High dimensional similarity search with space filling curves

A new class of geometric intersection searching problems is introduced, which generalizes previously-considered intersection searching problems and is rich in applications. In a standard intersection searching problem, a set S of n geometric objects is to be preprocessed so that the objects that are intersected by a query object q can be reported efficiently. In a generalized problem, the objects in S come aggregated in disjoint groups and what is of interest are the groups, not the objects, that are intersected by q. Although this problem can be solved easily by using an algorithm for the standard problem, the query time can be Ω(n) even though the output size is just O(1). In this paper, algorithms with efficient, output-size-sensitive query times are presented for the generalized versions of a number of intersection searching problems, including: interval intersection searching, orthogonal segment intersection searching, orthogonal range searching, point enclosure searching, rectangle intersection searching, and segment intersection searching. In addition, the algorithms are also space-efficient.

Mario A. Lopez

Papers

Indexing the positions of continuously moving objects

STR: a simple and efficient algorithm for R-tree packing

A greedy algorithm for bulk loading R-trees

High dimensional similarity search with space filling curves

Generalized intersection searching problems