Today's Web-enabled deluge of electronic data calls for automated methods of data analysis. Machine learning provides these, developing methods that can automatically detect patterns in data and then use the uncovered patterns to predict future data. This textbook offers a comprehensive and self-contained introduction to the field of machine learning, based on a unified, probabilistic approach. The coverage combines breadth and depth, offering necessary background material on such topics as probability, optimization, and linear algebra as well as discussion of recent developments in the field, including conditional random fields, L1 regularization, and deep learning. The book is written in an informal, accessible style, complete with pseudo-code for the most important algorithms. All topics are copiously illustrated with color images and worked examples drawn from such application domains as biology, text processing, computer vision, and robotics. Rather than providing a cookbook of different heuristic methods, the book stresses a principled model-based approach, often using the language of graphical models to specify models in a concise and intuitive way. Almost all the models described have been implemented in a MATLAB software package--PMTK (probabilistic modeling toolkit)--that is freely available online. The book is suitable for upper-level undergraduates with an introductory-level college math background and beginning graduate students.

Machine Learning : A Probabilistic Perspective

Statistics for Spatial Data.

Publishing data about individuals without revealing sensitive information about them is an important problem. In recent years, a new definition of privacy called k-anonymity has gained popularity. In a k-anonymized dataset, each record is indistinguishable from at least k − 1 other records with respect to certain identifying attributes.In this article, we show using two simple attacks that a k-anonymized dataset has some subtle but severe privacy problems. First, an attacker can discover the values of sensitive attributes when there is little diversity in those sensitive attributes. This is a known problem. Second, attackers often have background knowledge, and we show that k-anonymity does not guarantee privacy against attackers using background knowledge. We give a detailed analysis of these two attacks, and we propose a novel and powerful privacy criterion called e-diversity that can defend against such attacks. In addition to building a formal foundation for e-diversity, we show in an experimental evaluation that e-diversity is practical and can be implemented efficiently.

/pdf/l-diversity-privacy-beyond-k-anonymity-54qdj1526d.pdf

L-diversity: Privacy beyond k-anonymity

Publishing data about individuals without revealing sensitive information about them is an important problem. In recent years, a new definition of privacy called \kappa-anonymity has gained popularity. In a \kappa-anonymized dataset, each record is indistinguishable from at least k—1 other records with respect to certain "identifying" attributes. In this paper we show with two simple attacks that a \kappa-anonymized dataset has some subtle, but severe privacy problems. First, we show that an attacker can discover the values of sensitive attributes when there is little diversity in those sensitive attributes. Second, attackers often have background knowledge, and we show that \kappa-anonymity does not guarantee privacy against attackers using background knowledge. We give a detailed analysis of these two attacks and we propose a novel and powerful privacy definition called \ell-diversity. In addition to building a formal foundation for \ell-diversity, we show in an experimental evaluation that \ell-diversity is practical and can be implemented efficiently.

/pdf/l-diversity-privacy-beyond-k-anonymity-43op5i6gzs.pdf

L-diversity: privacy beyond k-anonymity

Discover New Methods for Dealing with High-Dimensional Data A sparse statistical model has only a small number of nonzero parameters or weights; therefore, it is much easier to estimate and interpret than a dense model. Statistical Learning with Sparsity: The Lasso and Generalizations presents methods that exploit sparsity to help recover the underlying signal in a set of data. Top experts in this rapidly evolving field, the authors describe the lasso for linear regression and a simple coordinate descent algorithm for its computation. They discuss the application of 1 penalties to generalized linear models and support vector machines, cover generalized penalties such as the elastic net and group lasso, and review numerical methods for optimization. They also present statistical inference methods for fitted (lasso) models, including the bootstrap, Bayesian methods, and recently developed approaches. In addition, the book examines matrix decomposition, sparse multivariate analysis, graphical models, and compressed sensing. It concludes with a survey of theoretical results for the lasso. In this age of big data, the number of features measured on a person or object can be large and might be larger than the number of observations. This book shows how the sparsity assumption allows us to tackle these problems and extract useful and reproducible patterns from big datasets. Data analysts, computer scientists, and theorists will appreciate this thorough and up-to-date treatment of sparse statistical modeling.

Statistical Learning with Sparsity: The Lasso and Generalizations

http://www.u.arizona.edu/~guangyao/papers/Dobra.JMA.2005.pdf

Sparse graphical models for exploring gene expression data

Despite the strikingly grave prognosis for older patients with glioblastomas, significant variability in patient outcome is experienced. To explore the potential for developing improved prognostic capabilities based on the elucidation of potential biological relationships, we did analyses of genes commonly mutated, amplified, or deleted in glioblastomas and DNA microarray gene expression data from tumors of glioblastoma patients of age >50 for whom survival is known. No prognostic significance was associated with genetic changes in epidermal growth factor receptor (amplified in 17 of 41 patients), TP53 (mutated in 11 of 41 patients), p16INK4A (deleted in 15 of 33 patients), or phosphatase and tensin homologue (mutated in 15 of 41 patients). Statistical analysis of the gene expression data in connection with survival involved exploration of regression models on small subsets of genes, based on computational search over multiple regression models with cross-validation to assess predictive validity. The analysis generated a set of regression models that, when weighted and combined according to posterior probabilities implied by the statistical analysis, identify patterns in expression of a small subset of genes that are associated with survival and have value in assessing survival risks. The dominant genes across such multiple regression models involve three key genes-SPARC (Osteonectin), Doublecortex, and Semaphorin3B-which play key roles in cellular migration processes. Additional analysis, based on statistical graphical association models constructed using similar computational analysis methods, reveals other genes which support the view that multiple mediators of tumor invasion may be important prognostic factor in glioblastomas in older patients.

https://cancerres.aacrjournals.org/content/canres/65/10/4051.full.pdf

Gene expression profiling and genetic markers in glioblastoma survival.

We discuss the implementation, development and performance of methods of stochastic computation in Gaussian graphical models. We view these methods from the perspective of high-dimensional model search, with a particular interest in the scalability with dimension of Markov chain Monte Carlo (MCMC) and other stochastic search methods. After reviewing the structure and context of undirected Gaussian graphical models and model uncertainty (covariance selection), we discuss prior specifications, including new priors over models, and then explore a number of examples using various methods of stochastic computation. Traditional MCMC methods are the point of departure for this experimentation; we then develop alternative stochastic search ideas and contrast this new approach with MCMC. Our examples range from low (12–20) to moderate (150) dimension, and combine simple synthetic examples with data analysis from gene expression studies. We conclude with comments about the need and potential for new computational methods in far higher dimensions, including constructive approaches to Gaussian graphical modeling and computation.

/pdf/experiments-in-stochastic-computation-for-high-dimensional-3adi3qk37v.pdf

Experiments in Stochastic Computation for High-Dimensional Graphical Models

Model search in regression with very large numbers of candidate predictors raises challenges for both model specification and computation, for which standard approaches such as Markov chain Monte Carlo (MCMC) methods are often infeasible or ineffective. We describe a novel shotgun stochastic search (SSS) approach that explores “interesting” regions of the resulting high-dimensional model spaces and quickly identifies regions of high posterior probability over models. We describe algorithmic and modeling aspects, priors over the model space that induce sparsity and parsimony over and above the traditional dimension penalization implicit in Bayesian and likelihood analyses, and parallel computation using cluster computers. We discuss an example from gene expression cancer genomics, comparisons with MCMC and other methods, and theoretical and simulation-based aspects of performance characteristics in large-scale regression model searches. We also provide software implementing the methods.

Shotgun Stochastic Search for “Large p” Regression

We propose a comprehensive Bayesian approach for graphical model determination in observational studies that can accommodate binary, ordinal or continuous variables simultaneously. Our new models are called copula Gaussian graphical models (CGGMs) and embed graphical model selection inside a semiparametric Gaussian copula. The domain of applicability of our methods is very broad and encompasses many studies from social science and economics. We illustrate the use of the copula Gaussian graphical models in the analysis of a 16-dimensional functional disability contingency table.

/pdf/copula-gaussian-graphical-models-and-their-application-to-2eoexzm6jq.pdf

Adrian Dobra

Papers

Sparse graphical models for exploring gene expression data

Gene expression profiling and genetic markers in glioblastoma survival.

Experiments in Stochastic Computation for High-Dimensional Graphical Models

Shotgun Stochastic Search for “Large p” Regression

Copula Gaussian graphical models and their application to modeling functional disability data