The growing interest in data mining is motivated by a common problem across disciplines: how does one store, access, model, and ultimately describe and understand very large data sets? Historically, different aspects of data mining have been addressed independently by different disciplines. This is the first truly interdisciplinary text on data mining, blending the contributions of information science, computer science, and statistics. The book consists of three sections. The first, foundations, provides a tutorial overview of the principles underlying data mining algorithms and their application. The presentation emphasizes intuition rather than rigor. The second section, data mining algorithms, shows how algorithms are constructed to solve specific problems in a principled manner. The algorithms covered include trees and rules for classification and regression, association rules, belief networks, classical statistical models, nonlinear models such as neural networks, and local "memory-based" models. The third section shows how all of the preceding analysis fits together when applied to real-world data mining problems. Topics include the role of metadata, how to handle missing data, and data preprocessing.

Principles of data mining

We present the Tiny AGgregation (TAG) service for aggregation in low-power, distributed, wireless environments. TAG allows users to express simple, declarative queries and have them distributed and executed efficiently in networks of low-power, wireless sensors. We discuss various generic properties of aggregates, and show how those properties affect the performance of our in network approach. We include a performance study demonstrating the advantages of our approach over traditional centralized, out-of-network methods, and discuss a variety of optimizations for improving the performance and fault tolerance of the basic solution.

/pdf/tag-a-tiny-aggregation-service-for-ad-hoc-sensor-networks-4ufv3hyh2r.pdf

TAG: a Tiny AGgregation service for Ad-Hoc sensor networks

The book Knowledge Discovery in Databases, edited by Piatetsky-Shapiro and Frawley [PSF91], is an early collection of research papers on knowledge discovery from data. The book Advances in Knowledge Discovery and Data Mining, edited by Fayyad, Piatetsky-Shapiro, Smyth, and Uthurusamy [FPSSe96], is a collection of later research results on knowledge discovery and data mining. There have been many data mining books published in recent years, including Predictive Data Mining by Weiss and Indurkhya [WI98], Data Mining Solutions: Methods and Tools for Solving Real-World Problems by Westphal and Blaxton [WB98], Mastering Data Mining: The Art and Science of Customer Relationship Management by Berry and Linofi [BL99], Building Data Mining Applications for CRM by Berson, Smith, and Thearling [BST99], Data Mining: Practical Machine Learning Tools and Techniques by Witten and Frank [WF05], Principles of Data Mining (Adaptive Computation and Machine Learning) by Hand, Mannila, and Smyth [HMS01], The Elements of Statistical Learning by Hastie, Tibshirani, and Friedman [HTF01], Data Mining: Introductory and Advanced Topics by Dunham, and Data Mining: Multimedia, Soft Computing, and Bioinformatics by Mitra and Acharya [MA03]. There are also books containing collections of papers on particular aspects of knowledge discovery, such as Machine Learning and Data Mining: Methods and Applications edited by Michalski, Brakto, and Kubat [MBK98], and Relational Data Mining edited by Dzeroski and Lavrac [De01], as well as many tutorial notes on data mining in major database, data mining and machine learning conferences.

/pdf/data-mining-concepts-and-techniques-2nd-edition-zx1xx1f5e3.pdf

Data Mining: Concepts and Techniques (2nd edition)

There is a growing need for ad-hoc analysis of extremely large data sets, especially at internet companies where innovation critically depends on being able to analyze terabytes of data collected every day. Parallel database products, e.g., Teradata, offer a solution, but are usually prohibitively expensive at this scale. Besides, many of the people who analyze this data are entrenched procedural programmers, who find the declarative, SQL style to be unnatural. The success of the more procedural map-reduce programming model, and its associated scalable implementations on commodity hardware, is evidence of the above. However, the map-reduce paradigm is too low-level and rigid, and leads to a great deal of custom user code that is hard to maintain, and reuse.We describe a new language called Pig Latin that we have designed to fit in a sweet spot between the declarative style of SQL, and the low-level, procedural style of map-reduce. The accompanying system, Pig, is fully implemented, and compiles Pig Latin into physical plans that are executed over Hadoop, an open-source, map-reduce implementation. We give a few examples of how engineers at Yahoo! are using Pig to dramatically reduce the time required for the development and execution of their data analysis tasks, compared to using Hadoop directly. We also report on a novel debugging environment that comes integrated with Pig, that can lead to even higher productivity gains. Pig is an open-source, Apache-incubator project, and available for general use.

/pdf/pig-latin-a-not-so-foreign-language-for-data-processing-1fxo6iweiv.pdf

Pig latin: a not-so-foreign language for data processing

The World Wide Web (WWW) continues to grow at an astounding rate in both the sheer volume of traffic and the size and complexity of Web sites. The complexity of tasks such as Web site design, Web server design, and of simply navigating through a Web site have increased along with this growth. An important input to these design tasks is the analysis of how a Web site is being used. Usage analysis includes straightforward statistics, such as page access frequency, as well as more sophisticated forms of analysis, such as finding the common traversal paths through a Web site. Web Usage Mining is the application of data mining techniques to usage logs of large Web data repositories in order to produce results that can be used in the design tasks mentioned above. However, there are several preprocessing tasks that must be performed prior to applying data mining algorithms to the data collected from server logs. This paper presents several data preparation techniques in order to identify unique users and user sessions. Also, a method to divide user sessions into semantically meaningful transactions is defined and successfully tested against two other methods. Transactions identified by the proposed methods are used to discover association rules from real world data using the WEBMINER system [15].

/pdf/data-preparation-for-mining-world-wide-web-browsing-patterns-x48kill86k.pdf

Data Preparation for Mining World Wide Web Browsing Patterns

Data analysis applications typically aggregate data across many dimensions looking for anomalies or unusual patterns. The SQL aggregate functions and the GROUP BY operator produce zero-dimensional or one-dimensional aggregates. Applications need the N-dimensional generalization of these operators. This paper defines that operator, called the data cube or simply cube. The cube operator generalizes the histogram, cross-tabulation, roll-up, drill-down, and sub-total constructs found in most report writers. The novelty is that cubes are relations. Consequently, the cube operator can be imbedded in more complex non-procedural data analysis programs. The cube operator treats each of the N aggregation attributes as a dimension of N-space. The aggregate of a particular set of attribute values is a point in this space. The set of points forms an N-dimensional cube. Super-aggregates are computed by aggregating the N-cube to lower dimensional spaces. This paper (1) explains the cube and roll-up operators, (2) shows how they fit in SQL, (3) explains how users can define new aggregate functions for cubes, and (4) discusses efficient techniques to compute the cube. Many of these features are being added to the SQL Standard.

Data Cube: A Relational Aggregation Operator Generalizing Group-By, Cross-Tab, and Sub-Totals

The disclosed declarative framework implements a machine for multi-step progression of interaction with an entity. The framework is usable for a broad range of applications—providing a simple rule-based authoring tool for specifying elements and components of a complex state machine, including state definitions, state transition triggers, state transition conditions and state transition actions. Case-status states, a first filtering condition, and a count parameter that specifies a limit on a number of times within a time period in excess of which additional events with characteristics that match will be ignored or discarded are usable to determine whether to ignore or process an incoming event—throttling the rate at which certain actions occur. A workflow engine gets loaded with instructions derived from the states and event filtering conditions, for handling incoming machine-generated events. Once defined, the state machine is automatically generated and implemented based on the declarative input provided.

Throttling Events in Entity Lifecycle Management

The technology disclosed offers a declarative framework that implements a machine for multi-step progression of interaction with an entity. The declarative framework is usable over and over for a broad range of applications because it provides a simple rule-based authoring tool that can be used for specifying different elements and components of a complex state machine, including state definitions, state transition triggers, state transition conditions and state transition actions. Once defined, the state machine is automatically generated and implemented based on the declarative input provided by a non-technical user.

Simplified entity lifecycle management

PROBLEM TO BE SOLVED: To simplify creation of an entity management workflow for non-programming users.SOLUTION: Provided is a method, comprising the steps of: displaying a plurality of case-status states in a workflow state machine that processes a machine generation event; storing in memory a first event filtering condition for determining whether to ignore or process an inputted second event during at least a first case-status state after a first event; initializing an integer count parameter, of the first filtering condition, for designating a limit to the number of times in a fixed period, with additional events exceeding the limit and having mutually matching features being ignored or abandoned; and loading an instruction for processing an inputted machine generation event to a workflow engine and executing the instruction selected in accordance with at least the case-status state and the first event filtering condition.SELECTED DRAWING: Figure 8A

Adam Bosworth

Papers

Data Cube: A Relational Aggregation Operator Generalizing Group-By, Cross-Tab, and Sub-Totals

Throttling Events in Entity Lifecycle Management

Simplified entity lifecycle management

State machine builder equipped with improved interface and processing of state independent event