Suppose x is an unknown vector in Ropfm (a digital image or signal); we plan to measure n general linear functionals of x and then reconstruct. If x is known to be compressible by transform coding with a known transform, and we reconstruct via the nonlinear procedure defined here, the number of measurements n can be dramatically smaller than the size m. Thus, certain natural classes of images with m pixels need only n=O(m1/4log5/2(m)) nonadaptive nonpixel samples for faithful recovery, as opposed to the usual m pixel samples. More specifically, suppose x has a sparse representation in some orthonormal basis (e.g., wavelet, Fourier) or tight frame (e.g., curvelet, Gabor)-so the coefficients belong to an lscrp ball for 0<ples1. The N most important coefficients in that expansion allow reconstruction with lscr2 error O(N1/2-1p/). It is possible to design n=O(Nlog(m)) nonadaptive measurements allowing reconstruction with accuracy comparable to that attainable with direct knowledge of the N most important coefficients. Moreover, a good approximation to those N important coefficients is extracted from the n measurements by solving a linear program-Basis Pursuit in signal processing. The nonadaptive measurements have the character of "random" linear combinations of basis/frame elements. Our results use the notions of optimal recovery, of n-widths, and information-based complexity. We estimate the Gel'fand n-widths of lscrp balls in high-dimensional Euclidean space in the case 0<ples1, and give a criterion identifying near- optimal subspaces for Gel'fand n-widths. We show that "most" subspaces are near-optimal, and show that convex optimization (Basis Pursuit) is a near-optimal way to extract information derived from these near-optimal subspaces

Compressed sensing

Today's Web-enabled deluge of electronic data calls for automated methods of data analysis. Machine learning provides these, developing methods that can automatically detect patterns in data and then use the uncovered patterns to predict future data. This textbook offers a comprehensive and self-contained introduction to the field of machine learning, based on a unified, probabilistic approach. The coverage combines breadth and depth, offering necessary background material on such topics as probability, optimization, and linear algebra as well as discussion of recent developments in the field, including conditional random fields, L1 regularization, and deep learning. The book is written in an informal, accessible style, complete with pseudo-code for the most important algorithms. All topics are copiously illustrated with color images and worked examples drawn from such application domains as biology, text processing, computer vision, and robotics. Rather than providing a cookbook of different heuristic methods, the book stresses a principled model-based approach, often using the language of graphical models to specify models in a concise and intuitive way. Almost all the models described have been implemented in a MATLAB software package--PMTK (probabilistic modeling toolkit)--that is freely available online. The book is suitable for upper-level undergraduates with an introductory-level college math background and beginning graduate students.

Machine Learning : A Probabilistic Perspective

Comprehensive Integration of Single-Cell Data.

We will review some of the major results in random graphs and some of the more challenging open problems. We will cover algorithmic and structural questions. We will touch on newer models, including those related to the WWW.

Random graphs

The lasso is a popular technique for simultaneous estimation and variable selection. Lasso variable selection has been shown to be consistent under certain conditions. In this work we derive a necessary condition for the lasso variable selection to be consistent. Consequently, there exist certain scenarios where the lasso is inconsistent for variable selection. We then propose a new version of the lasso, called the adaptive lasso, where adaptive weights are used for penalizing different coefficients in the l1 penalty. We show that the adaptive lasso enjoys the oracle properties; namely, it performs as well as if the true underlying model were given in advance. Similar to the lasso, the adaptive lasso is shown to be near-minimax optimal. Furthermore, the adaptive lasso can be solved by the same efficient algorithm for solving the lasso. We also discuss the extension of the adaptive lasso in generalized linear models and show that the oracle properties still hold under mild regularity conditions. As a bypro...

/pdf/the-adaptive-lasso-and-its-oracle-properties-p1nmst08nh.pdf

The adaptive lasso and its oracle properties

Most problems in frequentist statistics involve optimization of a function such as a likelihood or a sum of squares. EM algorithms are among the most effective algorithms for maximum likelihood estimation because they consistently drive the likelihood uphill by maximizing a simple surrogate function for the log-likelihood. Iterative optimization of a surrogate function as exemplified by an EM algorithm does not necessarily require missing data. Indeed, every EM algorithm is a special case of the more general class of MM optimization algorithms, which typically exploit convexity rather than missing data in majorizing or minorizing an objective function. In our opinion, MM algorithms deserve to be part of the standard toolkit of professional statisticians. This article explains the principle behind MM algorithms, suggests some methods for constructing them, and discusses some of their attractive features. We include numerous examples throughout the article to illustrate the concepts described. In addition t...

/pdf/a-tutorial-on-mm-algorithms-4yt18j1td5.pdf

A Tutorial on MM Algorithms

We describe some of the capabilities of the ergm package and the statistical theory underlying it. This package contains tools for accomplishing three important, and interrelated, tasks involving exponential-family random graph models (ERGMs): estimation, simulation, and goodness of t. More precisely, ergm has the capability of approximating a maximum likelihood estimator for an ERGM given a network data set; simulating new network data sets from a tted ERGM using Markov chain Monte Carlo; and assessing how well a tted ERGM does at capturing characteristics of a particular network data set.

/pdf/ergm-a-package-to-fit-simulate-and-diagnose-exponential-45kv69at5w.pdf

ergm: A Package to Fit, Simulate and Diagnose Exponential-Family Models for Networks.

The mixtools package for R provides a set of functions for analyzing a variety of nite mixture models. These functions include both traditional methods, such as EM algorithms for univariate and multivariate normal mixtures, and newer methods that reect some recent research in nite mixture models. In the latter category, mixtools provides algorithms for estimating parameters in a wide range of dierent mixture-of-regression contexts, in multinomial mixtures such as those arising from discretizing continuous multivariate data, in nonparametric situations where the multivariate component densities are completely unspecied, and in semiparametric situations such as a univariate location mixture of symmetric but otherwise unspecied densities. Many of the algorithms of the mixtools package are EM algorithms or are based on EM-like ideas, so this article includes an overview of EM algorithms for nite mixture models.

/pdf/mixtools-an-r-package-for-analyzing-finite-mixture-models-3h7t51z0lk.pdf

mixtools: An R Package for Analyzing Finite Mixture Models

The well-known EM algorithm is an optimization transfer algorithm that depends on the notion of incomplete or missing data By invoking convexity arguments, one can construct a variety of other optimization transfer algorithms that do not involve missing data These algorithms all rely on a majorizing or minorizing function that serves as a surrogate for the objective function Optimizing the surrogate function drives the objective function in the correct direction This article illustrates this general principle by a number of specific examples drawn from the statistical literature Because optimization transfer algorithms often exhibit the slow convergence of EM algorithms, two methods of accelerating optimization transfer are discussed and evaluated in the context of specific problems

http://lear.inrialpes.fr/people/oneata/reading_group/lange2000.pdf

Optimization Transfer Using Surrogate Objective Functions

statnet is a suite of software packages for statistical network analysis. The packages implement recent advances in network modeling based on exponential-family random graph models (ERGM). The components of the package provide a comprehensive framework for ERGM-based network modeling, including tools for model estimation, model evaluation, model-based network simulation, and network visualization. This broad functionality is powered by a central Markov chain Monte Carlo (MCMC) algorithm. The coding is optimized for speed and robustness.

/pdf/statnet-software-tools-for-the-representation-visualization-31eg07lkxh.pdf

David R. Hunter

Papers

A Tutorial on MM Algorithms

ergm: A Package to Fit, Simulate and Diagnose Exponential-Family Models for Networks.

mixtools: An R Package for Analyzing Finite Mixture Models

Optimization Transfer Using Surrogate Objective Functions

statnet: Software Tools for the Representation, Visualization, Analysis and Simulation of Network Data.