The increasing volume of data in modern business and science calls for more complex and sophisticated tools. Although advances in data mining technology have made extensive data collection much easier, it's still always evolving and there is a constant need for new techniques and tools that can help us transform this data into useful information and knowledge. Since the previous edition's publication, great advances have been made in the field of data mining. Not only does the third of edition of Data Mining: Concepts and Techniques continue the tradition of equipping you with an understanding and application of the theory and practice of discovering patterns hidden in large data sets, it also focuses on new, important topics in the field: data warehouses and data cube technology, mining stream, mining social networks, and mining spatial, multimedia and other complex data. Each chapter is a stand-alone guide to a critical topic, presenting proven algorithms and sound implementations ready to be used directly or with strategic modification against live data. This is the resource you need if you want to apply today's most powerful data mining techniques to meet real business challenges. * Presents dozens of algorithms and implementation examples, all in pseudo-code and suitable for use in real-world, large-scale data mining projects. * Addresses advanced topics such as mining object-relational databases, spatial databases, multimedia databases, time-series databases, text databases, the World Wide Web, and applications in several fields. *Provides a comprehensive, practical look at the concepts and techniques you need to get the most out of real business data

/pdf/data-mining-concepts-and-techniques-4dtvdfkvmi.pdf

Data Mining: Concepts and Techniques

Anomaly detection is an important problem that has been researched within diverse research areas and application domains. Many anomaly detection techniques have been specifically developed for certain application domains, while others are more generic. This survey tries to provide a structured and comprehensive overview of the research on anomaly detection. We have grouped existing techniques into different categories based on the underlying approach adopted by each technique. For each category we have identified key assumptions, which are used by the techniques to differentiate between normal and anomalous behavior. When applying a given technique to a particular domain, these assumptions can be used as guidelines to assess the effectiveness of the technique in that domain. For each category, we provide a basic anomaly detection technique, and then show how the different existing techniques in that category are variants of the basic technique. This template provides an easier and more succinct understanding of the techniques belonging to each category. Further, for each category, we identify the advantages and disadvantages of the techniques in that category. We also provide a discussion on the computational complexity of the techniques since it is an important issue in real application domains. We hope that this survey will provide a better understanding of the different directions in which research has been done on this topic, and how techniques developed in one area can be applied in domains for which they were not intended to begin with.

/pdf/anomaly-detection-a-survey-1x33g9m0a3.pdf

Anomaly detection: A survey

http://dtstc.ugr.es/~jedv/descargas/2009_CoSe09-Anomaly-based-network-intrusion-detection-Techniques,-systems-and-challenges.pdf

Anomaly-based network intrusion detection: Techniques, systems and challenges

An overview of anomaly detection techniques: Existing solutions and latest technological trends

With the increasing advances in hardware technology for data collection, and advances in software technology (databases) for data organization, computer scientists have increasingly participated in the latest advancements of the outlier analysis field. Computer scientists, specifically, approach this field based on their practical experiences in managing large amounts of data, and with far fewer assumptions the data can be of any type, structured or unstructured, and may be extremely large. Outlier Analysisis a comprehensive exposition, as understood by data mining experts, statisticians and computer scientists. The book has been organized carefully, and emphasis was placed on simplifying the content, so that students and practitioners can also benefit. Chapters will typically cover one of three areas: methods and techniques commonly used in outlier analysis, such as linear methods, proximity-based methods, subspace methods, and supervised methods; data domains, such as, text, categorical, mixed-attribute, time-series, streaming, discrete sequence, spatial and network data; and key applications of these methods as applied to diverse domains such as credit card fraud detection, intrusion detection, medical diagnosis, earth science, web log analytics, and social network analysis are covered.

Outlier Analysis

Intrusion detection complements prevention mechanisms, such as firewalls, cryptography, and authentication, to capture intrusions into an information system while they are acting on the information system. Our study investigates a multivariate quality control technique to detect intrusions by building a long-term profile of normal activities in information systems (norm profile) and using the norm profile to detect anomalies. The multivariate quality control technique is based on Hotelling's T/sup 2/ test that detects both counterrelationship anomalies and mean-shift anomalies. The performance of the Hotelling's T/sup 2/ test is examined on two sets of computer audit data: a small data set and a large multiday data set. Both data sets contain sessions of normal and intrusive activities. For the small data set, the Hotelling's T/sup 2/ test signals all the intrusion sessions and produces no false alarms for the normal sessions. For the large data set, the Hotelling's T/sup 2/ test signals 92 percent of the intrusion sessions while producing no false alarms for the normal sessions. The performance of the Hotelling's T/sup 2/ test is also compared with the performance of a more scalable multivariate technique-a chi-squared distance test.

/pdf/multivariate-statistical-analysis-of-audit-trails-for-host-4puz31h4mm.pdf

Multivariate statistical analysis of audit trails for host-based intrusion detection

An intrusion into an information system compromises its security (e.g. availability, integrity and confidentiality) through a series of events in the information system. Intrusive events often show departures (anomalies) from normal events in an information system. This paper presents an anomaly detection technique based on a chi-square statistic. This technique builds a profile of normal events in an information system—a norm profile computes the departure of events in the recent past from the norm profile and detects a large departure as an anomaly—a likely intrusion. This technique was tested for its performance in distinguishing normal events from intrusive events in an information system. The test results demonstrated the promising performance of this technique for intrusion detection in terms of a low false alarm rate and a high detection rate. Intrusive events were detected at a very early stage. Copyright © 2001 John Wiley & Sons, Ltd.

An anomaly detection technique based on a chi-square statistic for detecting intrusions into information systems

This paper presents a series of studies on probabilistic properties of activity data in an information system for detecting intrusions into the information system. Various probabilistic techniques of intrusion detection, including decision tree, Hotelling's T/sup 2/ test, chi-square multivariate test, and Markov chain are applied to the same training set and the same testing set of computer audit data for investigating the frequency property and the ordering property of computer audit data. The results of these studies provide answers to several questions concerning which properties are critical to intrusion detection. First, our studies show that the frequency property of multiple audit event types in a sequence of events is necessary for intrusion detection. A single audit event at a given time is not sufficient for intrusion detection. Second, the ordering property of multiple audit events provides additional advantage to the frequency property for intrusion detection. However, unless the scalability problem of complex data models taking into account the ordering property of activity data is solved, intrusion detection techniques based on the frequency property provide a viable solution that produces good intrusion detection performance with low computational overhead.

Probabilistic techniques for intrusion detection based on computer audit data

Reliability and quality of service from information systems has been threatened by cyber intrusions. To protect information systems from intrusions and thus assure reliability and quality of service, it is highly desirable to develop techniques that detect intrusions. Many intrusions manifest in anomalous changes in intensity of events occurring in information systems. In this study, we apply, test, and compare two EWMA techniques to detect anomalous changes in event intensity for intrusion detection: EWMA for autocorrelated data and EWMA for uncorrelated data. Different parameter settings and their effects on performance of these EWMA techniques are also investigated to provide guidelines for practical use of these techniques.

Computer intrusion detection through EWMA for autocorrelated and uncorrelated data

Intrusions into computer systems have caused many quality/reliability problems. Detecting intrusions is an important part of assuring the quality/reliability of computer systems by quickly detecting intrusions and associated quality/reliability problems in order to take corrective actions. In this paper, we present and compare two methods of forecasting normal activities in computer systems for intrusion detection. One forecasting method uses the average of long-term normal activities as the forecast. Another forecasting method uses the EWMA (exponentially weighted moving average) one-step-ahead forecast. We use a Markov chain model to learn and predict normal activities used in the EWMA forecasting method. A forecast of normal activities is used to detect a large deviation of the observed activities from the forecast as a possible intrusion into computer systems. A Chi square distance metric is used to measure the deviation of the observed activities from the forecast of normal activities. The two forecasting methods are tested on computer audit data of normal and intrusive activities for intrusion detection. The results indicate that the Chi square distance measure with the EWMA forecasting provides better performance in intrusion detection than that with the average-based forecasting method.

Qiang Chen

Papers

Multivariate statistical analysis of audit trails for host-based intrusion detection

An anomaly detection technique based on a chi-square statistic for detecting intrusions into information systems

Probabilistic techniques for intrusion detection based on computer audit data

Computer intrusion detection through EWMA for autocorrelated and uncorrelated data

EWMA forecast of normal system activity for computer intrusion detection