Data Mining - Concepts and Techniques.

Today's Web-enabled deluge of electronic data calls for automated methods of data analysis. Machine learning provides these, developing methods that can automatically detect patterns in data and then use the uncovered patterns to predict future data. This textbook offers a comprehensive and self-contained introduction to the field of machine learning, based on a unified, probabilistic approach. The coverage combines breadth and depth, offering necessary background material on such topics as probability, optimization, and linear algebra as well as discussion of recent developments in the field, including conditional random fields, L1 regularization, and deep learning. The book is written in an informal, accessible style, complete with pseudo-code for the most important algorithms. All topics are copiously illustrated with color images and worked examples drawn from such application domains as biology, text processing, computer vision, and robotics. Rather than providing a cookbook of different heuristic methods, the book stresses a principled model-based approach, often using the language of graphical models to specify models in a concise and intuitive way. Almost all the models described have been implemented in a MATLAB software package--PMTK (probabilistic modeling toolkit)--that is freely available online. The book is suitable for upper-level undergraduates with an introductory-level college math background and beginning graduate students.

Machine Learning : A Probabilistic Perspective

Statistics for Spatial Data.

Dynamic Bayesian Networks: Representation, Inference and Learning by Kevin Patrick Murphy Doctor of Philosophy in Computer Science University of California, Berkeley Professor Stuart Russell, Chair Modelling sequential data is important in many areas of science and engineering. Hidden Markov models (HMMs) and Kalman filter models (KFMs) are popular for this because they are simple and flexible. For example, HMMs have been used for speech recognition and bio-sequence analysis, and KFMs have been used for problems ranging from tracking planes and missiles to predicting the economy. However, HMMs and KFMs are limited in their “expressive power”. Dynamic Bayesian Networks (DBNs) generalize HMMs by allowing the state space to be represented in factored form, instead of as a single discrete random variable. DBNs generalize KFMs by allowing arbitrary probability distributions, not just (unimodal) linear-Gaussian. In this thesis, I will discuss how to represent many different kinds of models as DBNs, how to perform exact and approximate inference in DBNs, and how to learn DBN models from sequential data. In particular, the main novel technical contributions of this thesis are as follows: a way of representing Hierarchical HMMs as DBNs, which enables inference to be done in O(T ) time instead of O(T ), where T is the length of the sequence; an exact smoothing algorithm that takes O(log T ) space instead of O(T ); a simple way of using the junction tree algorithm for online inference in DBNs; new complexity bounds on exact online inference in DBNs; a new deterministic approximate inference algorithm called factored frontier; an analysis of the relationship between the BK algorithm and loopy belief propagation; a way of applying Rao-Blackwellised particle filtering to DBNs in general, and the SLAM (simultaneous localization and mapping) problem in particular; a way of extending the structural EM algorithm to DBNs; and a variety of different applications of DBNs. However, perhaps the main value of the thesis is its catholic presentation of the field of sequential data modelling.

/pdf/dynamic-bayesian-networks-representation-inference-and-yy4fhfxdcj.pdf

Dynamic bayesian networks: representation, inference and learning

Foreword Stephen P. Brooks, Andrew Gelman, Galin L. Jones, and Xiao-Li Meng Introduction to MCMC, Charles J. Geyer A short history of Markov chain Monte Carlo: Subjective recollections from in-complete data, Christian Robert and George Casella Reversible jump Markov chain Monte Carlo, Yanan Fan and Scott A. Sisson Optimal proposal distributions and adaptive MCMC, Jeffrey S. Rosenthal MCMC using Hamiltonian dynamics, Radford M. Neal Inference and Monitoring Convergence, Andrew Gelman and Kenneth Shirley Implementing MCMC: Estimating with confidence, James M. Flegal and Galin L. Jones Perfection within reach: Exact MCMC sampling, Radu V. Craiu and Xiao-Li Meng Spatial point processes, Mark Huber The data augmentation algorithm: Theory and methodology, James P. Hobert Importance sampling, simulated tempering and umbrella sampling, Charles J.Geyer Likelihood-free Markov chain Monte Carlo, Scott A. Sisson and Yanan Fan MCMC in the analysis of genetic data on related individuals, Elizabeth Thompson A Markov chain Monte Carlo based analysis of a multilevel model for functional MRI data, Brian Caffo, DuBois Bowman, Lynn Eberly, and Susan Spear Bassett Partially collapsed Gibbs sampling & path-adaptive Metropolis-Hastings in high-energy astrophysics, David van Dyk and Taeyoung Park Posterior exploration for computationally intensive forward models, Dave Higdon, C. Shane Reese, J. David Moulton, Jasper A. Vrugt and Colin Fox Statistical ecology, Ruth King Gaussian random field models for spatial data, Murali Haran Modeling preference changes via a hidden Markov item response theory model, Jong Hee Park Parallel Bayesian MCMC imputation for multiple distributed lag models: A case study in environmental epidemiology, Brian Caffo, Roger Peng, Francesca Dominici, Thomas A. Louis, and Scott Zeger MCMC for state space models, Paul Fearnhead MCMC in educational research, Roy Levy, Robert J. Mislevy, and John T. Behrens Applications of MCMC in fisheries science, Russell B. Millar Model comparison and simulation for hierarchical models: analyzing rural-urban migration in Thailand, Filiz Garip and Bruce Western

Handbook of Markov Chain Monte Carlo

Preface.1. Introduction.PART I: METHODOLOGY.2. Organisation of the data.3. Exploratory data analysis.4. Computational data mining.5. Statistical data mining.6. Evaluation of data mining methods.PART II: BUSINESS CASES.7. Market basket analysis.8. Web clickstream analysis.9. Profiling website visitors.10. Customer relationship management.11. Credit scoring.12. Forecasting television audience.Bibliography.Index.

Applied Data Mining: Statistical Methods for Business and Industry

The major implementational problem for reversible jump Markov chain Monte Carlo methods is that there is commonly no natural way to choose jump proposals since there is no Euclidean structure in the parameter space to guide our choice We consider mechanisms for guiding the choice of proposal The first group of methods is based on an analysis of acceptance probabilities for jumps Essentially, these methods involve a Taylor series expansion of the acceptance probability around certain canonical jumps and turn out to have close connections to Langevin algorithms The second group of methods generalizes the reversible jump algorithm by using the so-called saturated space approach These allow the chain to retain some degree of memory so that, when proposing to move from a smaller to a larger model, information is borrowed from the last time that the reverse move was performed The main motivation for this paper is that, in complex problems, the probability that the Markov chain moves between such spaces may be prohibitively small, as the probability mass can be very thinly spread across the space Therefore, finding reasonable jump proposals becomes extremely important We illustrate the procedure by using several examples of reversible jump Markov chain Monte Carlo applications including the analysis of autoregressive time series, graphical Gaussian modelling and mixture modelling

Efficient construction of reversible jump Markov chain Monte Carlo proposal distributions

We propose a methodology for Bayesian model determination in decomposable graphical Gaussian models. To achieve this aim we consider a hyper inverse Wishart prior distribution on the concentration matrix for each given graph. To ensure compatibility across models, such prior distributions are obtained by marginalisation from the prior conditional on the complete graph. We explore alternative structures for the hyperparameters of the latter, and their consequences for the model. Model determination is carried out by implementing a reversible jump Markov chain Monte Carlo sampler. In particular, the dimension-changing move we propose involves adding or dropping an edge from the graph. We characterise the set of moves which preserve the decomposability of the graph, giving a fast algorithm for maintaining the junction tree representation of the graph at each sweep. As state variable, we use the incomplete variance-covariance matrix, containing only the elements for which the corresponding element of the inverse is nonzero. This allows all computations to be performed locally, at the clique level, which is a clear advantage for the analysis of large and complex datasets. Finally, the statistical and computational performance of the procedure is illustrated by mean of both artificial and real datasets.

Decomposable graphical Gaussian model determination

The motivation of this paper is the application of MCMC model scoring procedures to data mining problems, involving a large number of competing models and other relevant model choice aspects. To achieve this aim we analyze one of the most popular Markov Chain Monte Carlo methods for structural learning in graphical models, namely, the MC
3 algorithm proposed by D. Madigan and J. York (International Statistical Review, 63, 215–232, 1995). Our aim is to improve their algorithm to make it an effective and reliable tool in the field of data mining. In such context, typically highly dimensional in the number of variables, little can be known a priori and, therefore, a good model search algorithm is crucial. We present and describe in detail our implementation of the MC
3 algorithm, which provides an efficient general framework for computations with both Directed Acyclic Graphical (DAG) models and Undirected Decomposable Models (UDG). We believe that the possibility of commuting easily between the two classes of models constitutes an important asset in data mining, where an a priori knowledge of causal effects is usually difficult to establish. Furthermore, in order to improve the MC
3 method we propose provide several graphical monitors which can help extracting results and assessing the goodness of the Markov chain Monte Carlo approximation to the posterior distribution of interest. We apply our proposed methodology first to the well-known coronary heart disease dataset (D. Edwards &; T. Havranek, Biometrika, 72:2, 339–351, 1985). We then introduce a novel data mining application which concerns market basket analysis.

/pdf/improving-markov-chain-monte-carlo-model-search-for-data-3t7af9m3ag.pdf

Improving Markov Chain Monte Carlo model search for data mining

1 Introduction. Part I Methodology. 2 Organisation of the data. 2.1 Statistical units and statistical variables. 2.2 Data matrices and their transformations. 2.3 Complex data structures. 2.4 Summary. 3 Summary statistics. 3.1 Univariate exploratory analysis. 3.1.1 Measures of location. 3.1.2 Measures of variability. 3.1.3 Measures of heterogeneity. 3.1.4 Measures of concentration. 3.1.5 Measures of asymmetry. 3.1.6 Measures of kurtosis. 3.2 Bivariate exploratory analysis of quantitative data. 3.3 Multivariate exploratory analysis of quantitative data. 3.4 Multivariate exploratory analysis of qualitative data. 3.4.1 Independence and association. 3.4.2 Distance measures. 3.4.3 Dependency measures. 3.4.4 Model-based measures. 3.5 Reduction of dimensionality. 3.5.1 Interpretation of the principal components. 3.6 Further reading. 4 Model specification. 4.1 Measures of distance. 4.1.1 Euclidean distance. 4.1.2 Similarity measures. 4.1.3 Multidimensional scaling. 4.2 Cluster analysis. 4.2.1 Hierarchical methods. 4.2.2 Evaluation of hierarchical methods. 4.2.3 Non-hierarchical methods. 4.3 Linear regression. 4.3.1 Bivariate linear regression. 4.3.2 Properties of the residuals. 4.3.3 Goodness of fit. 4.3.4 Multiple linear regression. 4.4 Logistic regression. 4.4.1 Interpretation of logistic regression. 4.4.2 Discriminant analysis. 4.5 Tree models. 4.5.1 Division criteria. 4.5.2 Pruning. 4.6 Neural networks. 4.6.1 Architecture of a neural network. 4.6.2 The multilayer perceptron. 4.6.3 Kohonen networks. 4.7 Nearest-neighbour models. 4.8 Local models. 4.8.1 Association rules. 4.8.2 Retrieval by content. 4.9 Uncertainty measures and inference. 4.9.1 Probability. 4.9.2 Statistical models. 4.9.3 Statistical inference. 4.10 Non-parametric modelling. 4.11 The normal linear model. 4.11.1 Main inferential results. 4.12 Generalised linear models. 4.12.1 The exponential family. 4.12.2 Definition of generalised linear models. 4.12.3 The logistic regression model. 4.13 Log-linear models. 4.13.1 Construction of a log-linear model. 4.13.2 Interpretation of a log-linear model. 4.13.3 Graphical log-linear models. 4.13.4 Log-linear model comparison. 4.14 Graphical models. 4.14.1 Symmetric graphical models. 4.14.2 Recursive graphical models. 4.14.3 Graphical models and neural networks. 4.15 Survival analysis models. 4.16 Further reading. 5 Model evaluation. 5.1 Criteria based on statistical tests. 5.1.1 Distance between statistical models. 5.1.2 Discrepancy of a statistical model. 5.1.3 Kullback-Leibler discrepancy. 5.2 Criteria based on scoring functions. 5.3 Bayesian criteria. 5.4 Computational criteria. 5.5 Criteria based on loss functions. 5.6 Further reading. Part II Business case studies. 6 Describing website visitors. 6.1 Objectives of the analysis. 6.2 Description of the data. 6.3 Exploratory analysis. 6.4 Model building. 6.4.1 Cluster analysis. 6.4.2 Kohonen networks. 6.5 Model comparison. 6.6 Summary report. 7 Market basket analysis. 7.1 Objectives of the analysis. 7.2 Description of the data. 7.3 Exploratory data analysis. 7.4 Model building. 7.4.1 Log-linear models. 7.4.2 Association rules. 7.5 Model comparison. 7.6 Summary report. 8 Describing customer satisfaction. 8.1 Objectives of the analysis. 8.2 Description of the data. 8.3 Exploratory data analysis. 8.4 Model building. 8.5 Summary. 9 Predicting credit risk of small businesses. 9.1 Objectives of the analysis. 9.2 Description of the data. 9.3 Exploratory data analysis. 9.4 Model building. 9.5 Model comparison. 9.6 Summary report. 10 Predicting e-learning student performance. 10.1 Objectives of the analysis. 10.2 Description of the data. 10.3 Exploratory data analysis. 10.4 Model specification. 10.5 Model comparison. 10.6 Summary report. 11 Predicting customer lifetime value. 11.1 Objectives of the analysis. 11.2 Description of the data. 11.3 Exploratory data analysis. 11.4 Model specification. 11.5 Model comparison. 11.6 Summary report. 12 Operational risk management. 12.1 Context and objectives of the analysis. 12.2 Exploratory data analysis. 12.3 Model building. 12.4 Model comparison. 12.5 Summary conclusions. References. Index.

Paolo Giudici

Papers

Applied Data Mining: Statistical Methods for Business and Industry

Efficient construction of reversible jump Markov chain Monte Carlo proposal distributions

Decomposable graphical Gaussian model determination

Improving Markov Chain Monte Carlo model search for data mining

Applied Data Mining for Business and Industry