Data Mining: Practical Machine Learning Tools and Techniques offers a thorough grounding in machine learning concepts as well as practical advice on applying machine learning tools and techniques in real-world data mining situations. This highly anticipated third edition of the most acclaimed work on data mining and machine learning will teach you everything you need to know about preparing inputs, interpreting outputs, evaluating results, and the algorithmic methods at the heart of successful data mining. Thorough updates reflect the technical changes and modernizations that have taken place in the field since the last edition, including new material on Data Transformations, Ensemble Learning, Massive Data Sets, Multi-instance Learning, plus a new version of the popular Weka machine learning software developed by the authors. Witten, Frank, and Hall include both tried-and-true techniques of today as well as methods at the leading edge of contemporary research. *Provides a thorough grounding in machine learning concepts as well as practical advice on applying the tools and techniques to your data mining projects *Offers concrete tips and techniques for performance improvement that work by transforming the input or output in machine learning methods *Includes downloadable Weka software toolkit, a collection of machine learning algorithms for data mining tasks-in an updated, interactive interface. Algorithms in toolkit cover: data pre-processing, classification, regression, clustering, association rules, visualization

Data Mining: Practical Machine Learning Tools and Techniques

List of Figures. List of Tables. Preface. Foreword. 1. Basic Concepts. 2. Evolutionary Algorithm MOP Approaches. 3. MOEA Test Suites. 4. MOEA Testing and Analysis. 5. MOEA Theory and Issues. 3. MOEA Theoretical Issues. 6. Applications. 7. MOEA Parallelization. 8. Multi-Criteria Decision Making. 9. Special Topics. 10. Epilog. Appendix A: MOEA Classification and Technique Analysis. Appendix B: MOPs in the Literature. Appendix C: Ptrue & PFtrue for Selected Numeric MOPs. Appendix D: Ptrue & PFtrue for Side-Constrained MOPs. Appendix E: MOEA Software Availability. Appendix F: MOEA-Related Information. Index. References.

/pdf/evolutionary-algorithms-for-solving-multi-objective-problems-2txxk4mqci.pdf

Evolutionary algorithms for solving multi-objective problems

General Strategies.- Regression Models.- Classification Models.- Other Considerations.- Appendix.- References.- Indices.

Applied Predictive Modeling

We evaluate 179 classifiers arising from 17 families (discriminant analysis, Bayesian, neural networks, support vector machines, decision trees, rule-based classifiers, boosting, bagging, stacking, random forests and other ensembles, generalized linear models, nearest-neighbors, partial least squares and principal component regression, logistic and multinomial regression, multiple adaptive regression splines and other methods), implemented in Weka, R (with and without the caret package), C and Matlab, including all the relevant classifiers available today. We use 121 data sets, which represent the whole UCI data base (excluding the large-scale problems) and other own real problems, in order to achieve significant conclusions about the classifier behavior, not dependent on the data set collection. The classifiers most likely to be the bests are the random forest (RF) versions, the best of which (implemented in R and accessed via caret) achieves 94.1% of the maximum accuracy overcoming 90% in the 84.3% of the data sets. However, the difference is not statistically significant with the second best, the SVM with Gaussian kernel implemented in C using LibSVM, which achieves 92.3% of the maximum accuracy. A few models are clearly better than the remaining ones: random forest, SVM with Gaussian and polynomial kernels, extreme learning machine with Gaussian kernel, C5.0 and avNNet (a committee of multi-layer perceptrons implemented in R with the caret package). The random forest is clearly the best family of classifiers (3 out of 5 bests classifiers are RF), followed by SVM (4 classifiers in the top-10), neural networks and boosting ensembles (5 and 3 members in the top-20, respectively).

/pdf/do-we-need-hundreds-of-classifiers-to-solve-real-world-1igtahh9y9.pdf

Do we need hundreds of classifiers to solve real world classification problems

Algorithms for feature selection fall into two broad categories: wrappers that use the learning algorithm itself to evaluate the usefulness of features and filters that evaluate features according to heuristics based on general characteristics of the data. For application to large databases, filters have proven to be more practical than wrappers because they are much faster. However, most existing filter algorithms only work with discrete classification problems. This paper describes a fast, correlation-based filter algorithm that can be applied to continuous and discrete problems. The algorithm often outperforms the well-known ReliefF attribute estimator when used as a preprocessing step for naive Bayes, instance-based learning, decision trees, locally weighted regression, and model trees. It performs more feature selection than ReliefF does—reducing the data dimensionality by fifty percent in most cases. Also, decision and model trees built from the preprocessed data are often significantly smaller.

Feature selection for discrete and numeric class machine learning

Induction of model trees for predicting continuous classes

Model trees, which are a type of decision tree with linear regression functions at the leaves, form the basis of a recent successful technique for predicting continuous numeric values. They can be applied to classification problems by employing a standard method of transforming a classification problem into a problem of function approximation. Surprisingly, using this simple transformation the model tree inducerM5 ′, based on Quinlan‘s M5, generates more accurate classifiers than the state-of-the-art decision tree learner C5.0, particularly when most of the attributes are numeric.

/pdf/using-model-trees-for-classification-ldonz4pesf.pdf

Using Model Trees for Classification

Summary. A fast algorithm for computing the non-parametric maximum likelihood estimate of a mixing distribution is presented. At each iteration, the algorithm adds new important points to the support set as guided by the gradient function, updates all mixing proportions via a quadratically convergent method and discards redundant support points straightaway. With its convergence being theoretically established, numerical studies show that it is very fast and stable, compared with several other algorithms that are available in the literature.

On fast computation of the non-parametric maximum likelihood estimate of a mixing distribution

We present a general modeling method for optimal probability prediction over future observations, in which model dimensionality is determined as a natural by-product. This new method yields several estimators, and we establish theoretically that they are optimal (either overall or under stated restrictions) when the number of free parameters is infinite. As a case study, we investigate the problem of fitting logistic models in finite-sample situations. Simulation results on both artificial and practical datasets are supportive.

Modeling for Optimal Probability Prediction

A new method is proposed for nonparametric multivariate density estimation, which extends a general framework that has been recently developed in the univariate case based on nonparametric and semiparametric mixture distributions. The major challenge to a multivariate extension is the dilemma that one can not maximise directly the likelihood function with respect to the whole component covariance matrix, since the likelihood is unbounded for a singular covariance matrix, and that one can not leave the covariance matrix or a large part of it to be determined by a selection method, since it would be computationally infeasible. To overcome it, we consider using a volume parameter h to enforce a minimal restriction on the covariance matrix so that, with h fixed, the likelihood function is bounded and its maximisation can be successfully carried out with respect to all the remaining parameters. The role played here by the scalar h is just the same as by the bandwidth in the univariate case and its value can be determined by a model selection criterion, such as the Akaike information criterion. New efficient algorithms are also described for finding the maximum likelihood estimates of these mixtures under various restrictions on the covariance matrix. Empirical studies using simulated and real-world data show that the new multivariate mixture-based density estimator performs remarkably better than kernel-based density estimators.

Yong Wang

Papers

Induction of model trees for predicting continuous classes

Using Model Trees for Classification

On fast computation of the non-parametric maximum likelihood estimate of a mixing distribution

Modeling for Optimal Probability Prediction

Nonparametric multivariate density estimation using mixtures