Data Mining Practical Machine Learning Tools and Techniques

We explore the effect of dimensionality on the nearest neighbor problem. We show that under a broad set of conditions (much broader than independent and identically distributed dimensions), as dimensionality increases, the distance to the nearest data point approaches the distance to the farthest data point. To provide a practical perspective, we present empirical results on both real and synthetic data sets that demonstrate that this effect can occur for as few as 10-15 dimensions. These results should not be interpreted to mean that high-dimensional indexing is never meaningful; we illustrate this point by identifying some high-dimensional workloads for which this effect does not occur. However, our results do emphasize that the methodology used almost universally in the database literature to evaluate high-dimensional indexing techniques is flawed, and should be modified. In particular, most such techniques proposed in the literature are not evaluated versus simple linear scan, and are evaluated over workloads for which nearest neighbor is not meaningful. Often, even the reported experiments, when analyzed carefully, show that linear scan would outperform the techniques being proposed on the workloads studied in high (10-15) dimensionality!.

When is nearest neighbor meaningful

The Visual Display of Quantitative Information

Urbanization's rapid progress has modernized many people's lives but also engendered big issues, such as traffic congestion, energy consumption, and pollution. Urban computing aims to tackle these issues by using the data that has been generated in cities (e.g., traffic flow, human mobility, and geographical data). Urban computing connects urban sensing, data management, data analytics, and service providing into a recurrent process for an unobtrusive and continuous improvement of people's lives, city operation systems, and the environment. Urban computing is an interdisciplinary field where computer sciences meet conventional city-related fields, like transportation, civil engineering, environment, economy, ecology, and sociology in the context of urban spaces. This article first introduces the concept of urban computing, discussing its general framework and key challenges from the perspective of computer sciences. Second, we classify the applications of urban computing into seven categories, consisting of urban planning, transportation, the environment, energy, social, economy, and public safety and security, presenting representative scenarios in each category. Third, we summarize the typical technologies that are needed in urban computing into four folds, which are about urban sensing, urban data management, knowledge fusion across heterogeneous data, and urban data visualization. Finally, we give an outlook on the future of urban computing, suggesting a few research topics that are somehow missing in the community.

/pdf/urban-computing-concepts-methodologies-and-applications-1wix0tsb38.pdf

Urban Computing: Concepts, Methodologies, and Applications

Visual analytics aims to combine the strengths of human and electronic data processing. Visualisation, whereby humans and computers cooperate through graphics, is the means through which this is achieved. Seamless and sophisticated synergies are required for analysing spatio-temporal data and solving spatio-temporal problems. In modern society, spatio-temporal analysis is not solely the business of professional analysts. Many citizens need or would be interested in undertaking analysis of information in time and space. Researchers should find approaches to deal with the complexities of the current data and problems and find ways to make analytical tools accessible and usable for the broad community of potential users to support spatio-temporal thinking and contribute to solving a large range of problems.

/pdf/space-time-and-visual-analytics-4e1ge8zd15.pdf

Space, time and visual analytics

Spatiotemporal data pose serious challenges to analysts in geographic and other domains. Owing to the complexity of the geospatial and temporal components, this kind of data cannot be analyzed by fully automatic methods but require the involvement of the human analyst's expertise. For a comprehensive analysis, the data need to be considered from two complementary perspectives: (1) as spatial distributions (situations) changing over time and (2) as profiles of local temporal variation distributed over space. In order to support the visual analysis of spatiotemporal data, we suggest a framework based on the "Self-Organizing Map" (SOM) method combined with a set of interactive visual tools supporting both analytic perspectives. SOM can be considered as a combination of clustering and dimensionality reduction. In the first perspective, SOM is applied to the spatial situations at different time moments or intervals. In the other perspective, SOM is applied to the local temporal evolution profiles. The integrated visual analytics environment includes interactive coordinated displays enabling various transformations of spatiotemporal data and post-processing of SOM results. The SOM matrix display offers an overview of the groupings of data objects and their two-dimensional arrangement by similarity. This view is linked to a cartographic map display, a time series graph, and a periodic pattern view. The linkage of these views supports the analysis of SOM results in both the spatial and temporal contexts. The variable SOM grid coloring serves as an instrument for linking the SOM with the corresponding items in the other displays. The framework has been validated on a large dataset with real city traffic data, where expected spatiotemporal patterns have been successfully uncovered. We also describe the use of the framework for discovery of previously unknown patterns in 41-years time series of 7 crime rate attributes in the states of the USA.

/pdf/space-in-time-and-time-in-space-self-organizing-maps-for-1bm0ak5z32.pdf

Space-in-time and time-in-space self-organizing maps for exploring spatiotemporal patterns

Traditionally, the visual analysis of hierarchies, respectively, trees, is conducted by focusing on one given hierarchy. However, in many research areas multiple, differing hierarchies need to be analyzed simultaneously in a comparative way - in particular to highlight differences between them, which sometimes can be subtle. A prominent example is the analysis of so-called phylogenetic trees in biology, reflecting hierarchical evolutionary relationships among a set of organisms. Typically, the analysis considers multiple phylogenetic trees, either to account for statistical significance or for differences in derivation of such evolutionary hierarchies; for example, based on different input data, such as the 16S ribosomal RNA and protein sequences of highly conserved enzymes. The simultaneous analysis of a collection of such trees leads to more insight into the evolutionary process. We introduce a novel visual analytics approach for the comparison of multiple hierarchies focusing on both global and local structures. A new tree comparison score has been elaborated for the identification of interesting patterns. We developed a set of linked hierarchy views showing the results of automatic tree comparison on various levels of details. This combined approach offers detailed assessment of local and global tree similarities. The approach was developed in close cooperation with experts from the evolutionary biology domain. We apply it to a phylogenetic data set on bacterial ancestry, demonstrating its application benefit.

/pdf/interactive-visual-comparison-of-multiple-trees-2olcd1i40x.pdf

Interactive visual comparison of multiple trees

The analysis of high-dimensional data is an important, yet inherently difficult problem. Projection techniques such as Principal Component Analysis, Multi-dimensional Scaling and Self-Organizing Map can be used to map high-dimensional data to 2D display space. However, projections typically incur a loss in information. Often, uncertainty exists regarding the precision of the projection as compared with its original data characteristics. While the output quality of these projection techniques can be discussed in terms of aggregate numeric error values, visualization is often helpful for better understanding the projection results. We address the visual assessment of projection precision by an approach integrating an appropriately designed projection precision measure directly into the projection visualization. To this end, a flexible projection precision measure is defined that allows the user to balance the degree of locality at which the measure is evaluated. Several visual mappings are designed for integrating the precision measure into the projection visualization at various levels of abstraction. The techniques are implemented in an interactive system, including methods supporting the user in finding appropriate settings of relevant parameters. We demonstrate the usefulness of the approach for visual analysis of classified and unclassified high-dimensional data sets. We show how our interactive precision quality visualization system helps to examine the preservation of original data properties in projected space.

/pdf/techniques-for-precision-based-visual-analysis-of-projected-b1gkppnbmi.pdf

Techniques for precision-based visual analysis of projected data

We focus on visual analysis of space- and time-referenced categorical data, which describe possible states of spatial (geographical) objects or locations and their changes over time. The analysis of these data is difficult as there are only limited possibilities to analyze the three aspects (location, time and category) simultaneously. We present a new approach which interactively combines (a) visualization of categorical changes over time; (b) various spatial data displays; (c) computational techniques for task-oriented selection of time steps. They provide an expressive visualization with regard to either the overall evolution over time or unusual changes. We apply our approach on two use cases demonstrating its usefulness for a wide variety of tasks. We analyze data from movement tracking and meteorologic areas. Using our approach, expected events could be detected and new insights were gained.

Visual analytics methods for categoric spatio-temporal data

Exploration and selection of data descriptors representing objects using a set of features are important components in many data analysis tasks. Usually, for a given dataset, an optimal data description does not exist, as the suitable data representation is strongly use case dependent. Many solutions for selecting a suitable data description have been proposed. In most instances, they require data labels and often are black box approaches. Non-expert users have difficulties to comprehend the coherency of input, parameters, and output of these algorithms. Alternative approaches, interactive systems for visual feature selection, overburden the user with an overwhelming set of options and data views. Therefore, it is essential to offer the users a guidance in this analytical process.

In this paper, we present a novel system for data description selection, which facilitates the user's access to the data analysis process. As finding of suitable data description consists of several steps, we support the user with guidance. Our system combines automatic data analysis with interactive visualizations. By this, the system provides a recommendation for suitable data descriptor selections. It supports the comparison of data descriptors with differing dimensionality for unlabeled data. We propose specialized scores and interactive views for descriptor comparison. The visualization techniques are scatterplot-based and grid-based. For the latter case, we apply Self-Organizing Maps as adaptive grids which are well suited for large multi-dimensional data sets. As an example, we demonstrate the usability of our system on a real-world biochemical application.

/pdf/assisted-descriptor-selection-based-on-visual-comparative-2sxtu6dsvn.pdf

Sebastian Bremm

Papers

Space-in-time and time-in-space self-organizing maps for exploring spatiotemporal patterns

Interactive visual comparison of multiple trees

Techniques for precision-based visual analysis of projected data

Visual analytics methods for categoric spatio-temporal data

Assisted descriptor selection based on visual comparative data analysis