We explore the effect of dimensionality on the nearest neighbor problem. We show that under a broad set of conditions (much broader than independent and identically distributed dimensions), as dimensionality increases, the distance to the nearest data point approaches the distance to the farthest data point. To provide a practical perspective, we present empirical results on both real and synthetic data sets that demonstrate that this effect can occur for as few as 10-15 dimensions. These results should not be interpreted to mean that high-dimensional indexing is never meaningful; we illustrate this point by identifying some high-dimensional workloads for which this effect does not occur. However, our results do emphasize that the methodology used almost universally in the database literature to evaluate high-dimensional indexing techniques is flawed, and should be modified. In particular, most such techniques proposed in the literature are not evaluated versus simple linear scan, and are evaluated over workloads for which nearest neighbor is not meaningful. Often, even the reported experiments, when analyzed carefully, show that linear scan would outperform the techniques being proposed on the workloads studied in high (10-15) dimensionality!.

When is nearest neighbor meaningful

Search operations in databases require special support at the physical level. This is true for conventional databases as well as spatial databases, where typical search operations include the point query (find all objects that contain a given search point) and the region query (find all objects that overlap a given search region). More than ten years of spatial database research have resulted in a great variety of multidimensional access methods to support such operations. We give an overview of that work. After a brief survey of spatial data management in general, we first present the class of point access methods, which are used to search sets of points in two or more dimensions. The second part of the paper is devoted to spatial access methods to handle extended objects, such as rectangles or polyhedra. We conclude with a discussion of theoretical and experimental results concerning the relative performance of various approaches.

/pdf/multidimensional-access-methods-4jxurm5el0.pdf

Multidimensional access methods

Existing trajectory clustering algorithms group similar trajectories as a whole, thus discovering common trajectories. Our key observation is that clustering trajectories as a whole could miss common sub-trajectories. Discovering common sub-trajectories is very useful in many applications, especially if we have regions of special interest for analysis. In this paper, we propose a new partition-and-group framework for clustering trajectories, which partitions a trajectory into a set of line segments, and then, groups similar line segments together into a cluster. The primary advantage of this framework is to discover common sub-trajectories from a trajectory database. Based on this partition-and-group framework, we develop a trajectory clustering algorithm TRACLUS. Our algorithm consists of two phases: partitioning and grouping. For the first phase, we present a formal trajectory partitioning algorithm using the minimum description length(MDL) principle. For the second phase, we present a density-based line-segment clustering algorithm. Experimental results demonstrate that TRACLUS correctly discovers common sub-trajectories from real trajectory data.

/pdf/trajectory-clustering-a-partition-and-group-framework-2b61tdpjnn.pdf

Trajectory clustering: a partition-and-group framework

The last decade has witnessed a tremendous growths of interests in applications that deal with querying and mining of time series data. Numerous representation methods for dimensionality reduction and similarity measures geared towards time series have been introduced. Each individual work introducing a particular method has made specific claims and, aside from the occasional theoretical justifications, provided quantitative experimental observations. However, for the most part, the comparative aspects of these experiments were too narrowly focused on demonstrating the benefits of the proposed methods over some of the previously introduced ones. In order to provide a comprehensive validation, we conducted an extensive set of time series experiments re-implementing 8 different representation methods and 9 similarity measures and their variants, and testing their effectiveness on 38 time series data sets from a wide variety of application domains. In this paper, we give an overview of these different techniques and present our comparative experimental findings regarding their effectiveness. Our experiments have provided both a unified validation of some of the existing achievements, and in some cases, suggested that certain claims in the literature may be unduly optimistic.

/pdf/querying-and-mining-of-time-series-data-experimental-4uzriiubdj.pdf

Querying and mining of time series data: experimental comparison of representations and distance measures

A review on time series data mining

Applications in virtual multimedia catalogs are highly interactive. Thus, it is difficult to estimate resource demands required for presentation of catalog contents. We propose a method to predict presentation resource demands in interactive multimedia catalogs. The prediction is based on the results of mining the virtual mall action log file. The log file typically contains information about previous user interests and browsing behavior. These data are used for modeling users future behavior within a session. We define heuristics to generate a start-up user behavior model as a continuous time Markov chain and adapt this model during a running session to the current user.

/pdf/mining-user-behavior-for-resource-prediction-in-interactive-1b25ayrjjb.pdf

Mining user behavior for resource prediction in interactive electronic malls

Spatial joins are join operations that involve spatial data types and operators. Spatial access methods are often used to speed up the computation of spatial joins. This paper addresses the issue of benchmarking spatial join operations. For this purpose, we first present a WWW-based benchmark generator to produce sets of rectangles. Using a Web browser, experimenters can specify the number of rectangles in a sample, as well as the statistical distributions of their sizes, shapes, and locations. Second, using the generator and a well-defined set of statistical models we define several tests to compare the performance of three spatial join algorithms: nested loop, scan-and-index, and synchronized tree traversal. We also added two real-life data sets from the Sequoia 2000 storage benchmark. Our results show that the relative performance of the different techniques mainly depends on the selectivity factor of the join predicate. All of the statistical models and algorithms are available on the Web, which allow...

Benchmarking spatial joins a la carte

Web search engines have become the necessary tools for Web users to search useful information However, users often face problems such as how to find the right information promptly with the least effort This paper examines the limitations of current Web search engines and proposes a new Web meta-search engine architecture The proposed architecture uses a multi-agent approach to process the user queries with greater personalization functionality and higher result quality than regular meta-search engines Thus, the propose method improves the usability and efficiency of current search engines

/pdf/a-new-architecture-for-web-meta-search-engines-494d74p38j.pdf

A New Architecture for Web Meta-Search Engines

In the U.S., 80% of Medicare spending is for managing patients with multiple coexisting conditions. Predicting potentially correlated diseases for an individual patient and correlated disease progression paths are both important research tasks. For example, obese patients are at an increased risk for developing type-2 diabetes and hypertension. This correlation is called comorbidity relationship. Discovering the comorbidity relationships is complex and difficult due to the limited access to Electronic Health Records by privacy laws. In this paper, we present a framework called Social Data-based Prediction of Incidence and Trajectory to predict potential risks for medical conditions as well as its progression trajectory to identify the comorbidity path. The framework utilizes patients' publicly available social media data and presents a collaborative prediction model to predict the ranked list of potential comorbidity incidences, and a trajectory prediction model to reveal different paths of condition progression. The experimental results show that our framework is able to predict future conditions for online patients with a coverage value of 48% and 75% for a top-20 and a top-100 ranked list, respectively. For risk trajectory prediction, our framework is able to reveal each potential progression trajectory between any two conditions and infer the confidence of the future trajectory, given any observed condition. The predicted trajectories are validated with existing comorbidity relations from the medical literature.

Collaborative and trajectory prediction models of medical conditions by mining patients' Social Data

Many applications — such as content-based image retrieval, subspace clustering, and feature selection — may benefit from efficient subspace similarity search. Given a query object, the goal of subspace similarity search is to retrieve the most similar objects from the database, where the similarity distance is defined over an arbitrary subset of dimensions (or features) — that is, an arbitrary axis-aligned projective subspace. Though much effort has been spent on similarity search in fixed subspaces, relatively little attention has been given to the problem of similarity search when the dimensions are specified at query time. In this paper, we propose several new methods for the subspace similarity search problem. Extensive experiments are provided showing very competitive performance relative to state-of-the-art solutions.

Vincent Oria

Papers

Mining user behavior for resource prediction in interactive electronic malls

Benchmarking spatial joins a la carte

A New Architecture for Web Meta-Search Engines

Collaborative and trajectory prediction models of medical conditions by mining patients' Social Data

Efficient Algorithms for Similarity Search in Axis-Aligned Subspaces