Data Mining - Concepts and Techniques.

We explore the effect of dimensionality on the nearest neighbor problem. We show that under a broad set of conditions (much broader than independent and identically distributed dimensions), as dimensionality increases, the distance to the nearest data point approaches the distance to the farthest data point. To provide a practical perspective, we present empirical results on both real and synthetic data sets that demonstrate that this effect can occur for as few as 10-15 dimensions. These results should not be interpreted to mean that high-dimensional indexing is never meaningful; we illustrate this point by identifying some high-dimensional workloads for which this effect does not occur. However, our results do emphasize that the methodology used almost universally in the database literature to evaluate high-dimensional indexing techniques is flawed, and should be modified. In particular, most such techniques proposed in the literature are not evaluated versus simple linear scan, and are evaluated over workloads for which nearest neighbor is not meaningful. Often, even the reported experiments, when analyzed carefully, show that linear scan would outperform the techniques being proposed on the workloads studied in high (10-15) dimensionality!.

When is nearest neighbor meaningful

The Visual Display of Quantitative Information

Computational geometry

The key to utilizing the potential of multirobot systems is cooperation. How can we achieve cooperation in systems composed of failure-prone autonomous robots operating in noisy, dynamic environments? We present a method of dynamic task allocation for groups of such robots. We implemented and tested an auction-based task allocation system which we call MURDOCH, built upon a principled, resource centric, publish/subscribe communication model. A variant of the Contract Net Protocol, MURDOCH produces a distributed approximation to a global optimum of resource usage. We validated MURDOCH in two very different domains: a tightly coupled multirobot physical manipulation task and a loosely coupled multirobot experiment in long-term autonomy. The primary contribution of the paper is to show empirically that distributed negotiation mechanisms such as MURDOCH are viable and effective for coordinating physical multirobot systems.

/pdf/sold-auction-methods-for-multirobot-coordination-4kyen6pxfc.pdf

Sold!: auction methods for multirobot coordination

Data warehousing systems enable enterprise managers to acquire and integrate information from heterogeneous sources and to query very large databases efficiently. Building a data warehouse requires adopting design and implementation techniques completely different from those underlying operational information systems. Though most scientific literature on the design of data warehouses concerns their logical and physical models, an accurate conceptual design is the necessary foundations for building a DW which is well-documented and fully satisfies requirements. In this paper we formalize a graphical conceptual model for data warehouses, called Dimensional Fact model, and propose a semi-automated methodology to build it from the pre-existing (conceptual or logical) schemes describing the enterprise relational database. The representation of reality built using our conceptual model consists of a set of fact schemes whose basic elements are facts, measures, attributes, dimensions and hierarchies; other featur...

The dimensional fact model: a conceptual model for data warehouses

During the last ten years the approach to business management has deeply changed, and companies have understood the importance of enforcing achievement of the goals defined by their strategy through metrics-driven management. The DW process, though supporting bottom-up extraction of information from data, fails in top-down enforcing the company strategy. A new approach to BI, called Business Performance Management (BPM), is emerging from this framework: it includes DW but it also requires a reactive component capable of monitoring the time-critical operational processes to allow tactical and operational decision-makers to tune their actions according to the company strategy. The aim of this paper is to encourage the research community to acknowledge the coming of a second era in BI, to propose a general architecture for BPM, and to lay the premises for investigating the most challenging of the related issues.

/pdf/beyond-data-warehousing-what-s-next-in-business-intelligence-3ds6mde3hw.pdf

Beyond data warehousing: what's next in business intelligence?

Data warehousing systems enable enterprise managers to acquire and integrate information from heterogeneous sources and to query very large databases efficiently. Building a data warehouse requires adopting design and implementation techniques completely different from those underlying information systems. We present a graphical conceptual model for data warehouses, called Dimensional Fact model, and propose a semi-automated methodology to build it from the pre-existing entity/relationship schemes describing a database. Our conceptual model consists of tree-structured fact schemes whose basic elements are facts, attributes, dimensions and hierarchies; other features which may be represented on fact schemes are the additivity of fact attributes along dimensions, the optionality of dimension attributes and the existence of non-dimension attributes. Compatible fact schemes may be overlapped in order to relate and compare data. Fact schemes may be integrated with information of the conjectured workload, expressed in terms of query patterns, to be used as the input of a design phase whose output are the logical and physical schemes of the data warehouse.

/pdf/conceptual-design-of-data-warehouses-from-e-r-schemes-2coohyulsj.pdf

Conceptual design of data warehouses from E/R schemes

Chapter 1. Introduction to Data Warehousing Chapter 2. Data Warehouse System Lifecycle Chapter 3. Analysis and Reconciliation of Data Sources Chapter 4. User Requirement Analysis Chapter 5. Conceptual Modeling Chapter 6. Conceptual Design Chapter 7. Workload and Data Volume Chapter 8. Logical Modeling Chapter 9. Logical Design Chapter 10. Data-staging Design Chapter 11. Indexes for the Data Warehouse Chapter 12. Physical Design Chapter 13. Data Warehouse Project Documentation Chapter 14. A Case Study Chapter 15. Business Intelligence: Beyond the Data Warehouse Glossary Bibliography Index

Data Warehouse Design: Modern Principles and Methodologies

Though designing a data warehouse requires techniques completely different from those adopted for operational systems, no significant effort has been made so far to develop a complete and consistent design methodology for data warehouses. In this paper we outline a general methodological framework for data warehouse design, based on our Dimensional Fact Model (DFM). After analyzing the existing information system and collecting the user requirements, conceptual design is carried out semi-automatically starting from the operational database scheme. A workload is then characterized in terms of data volumes and expected queries, to be used as the input of the logical and physical design phases whose output is the final scheme for the data warehouse.

/pdf/a-methodological-framework-for-data-warehouse-design-4287m9iuet.pdf

Stefano Rizzi

Papers

The dimensional fact model: a conceptual model for data warehouses

Beyond data warehousing: what's next in business intelligence?

Conceptual design of data warehouses from E/R schemes

Data Warehouse Design: Modern Principles and Methodologies

A methodological framework for data warehouse design