Today's Web-enabled deluge of electronic data calls for automated methods of data analysis. Machine learning provides these, developing methods that can automatically detect patterns in data and then use the uncovered patterns to predict future data. This textbook offers a comprehensive and self-contained introduction to the field of machine learning, based on a unified, probabilistic approach. The coverage combines breadth and depth, offering necessary background material on such topics as probability, optimization, and linear algebra as well as discussion of recent developments in the field, including conditional random fields, L1 regularization, and deep learning. The book is written in an informal, accessible style, complete with pseudo-code for the most important algorithms. All topics are copiously illustrated with color images and worked examples drawn from such application domains as biology, text processing, computer vision, and robotics. Rather than providing a cookbook of different heuristic methods, the book stresses a principled model-based approach, often using the language of graphical models to specify models in a concise and intuitive way. Almost all the models described have been implemented in a MATLAB software package--PMTK (probabilistic modeling toolkit)--that is freely available online. The book is suitable for upper-level undergraduates with an introductory-level college math background and beginning graduate students.

Machine Learning : A Probabilistic Perspective

/pdf/convex-analysisnoer-san-nojin-zhan-nituite-5dl2rbmoyr.pdf

Convex Analysisの二,三の進展について

This monograph is about a class of optimization algorithms called proximal algorithms. Much like Newton's method is a standard tool for solving unconstrained smooth optimization problems of modest size, proximal algorithms can be viewed as an analogous tool for nonsmooth, constrained, large-scale, or distributed versions of these problems. They are very generally applicable, but are especially well-suited to problems of substantial recent interest involving large or high-dimensional datasets. Proximal methods sit at a higher level of abstraction than classical algorithms like Newton's method: the base operation is evaluating the proximal operator of a function, which itself involves solving a small convex optimization problem. These subproblems, which generalize the problem of projecting a point onto a convex set, often admit closed-form solutions or can be solved very quickly with standard or simple specialized methods. Here, we discuss the many different interpretations of proximal operators and algorithms, describe their connections to many other topics in optimization and applied mathematics, survey some popular algorithms, and provide a large number of examples of proximal operators that commonly arise in practice.

Proximal Algorithms

Compressed sensing aims to undersample certain high-dimensional signals yet accurately reconstruct them by exploiting signal characteristics. Accurate reconstruction is possible when the object to be recovered is sufficiently sparse in a known basis. Currently, the best known sparsity–undersampling tradeoff is achieved when reconstructing by convex optimization, which is expensive in important large-scale applications. Fast iterative thresholding algorithms have been intensively studied as alternatives to convex optimization for large-scale problems. Unfortunately known fast algorithms offer substantially worse sparsity–undersampling tradeoffs than convex optimization. We introduce a simple costless modification to iterative thresholding making the sparsity–undersampling tradeoff of the new algorithms equivalent to that of the corresponding convex optimization procedures. The new iterative-thresholding algorithms are inspired by belief propagation in graphical models. Our empirical measurements of the sparsity–undersampling tradeoff for the new algorithms agree with theoretical calculations. We show that a state evolution formalism correctly derives the true sparsity–undersampling tradeoff. There is a surprising agreement between earlier calculations based on random convex polytopes and this apparently very different theoretical formalism.

/pdf/message-passing-algorithms-for-compressed-sensing-29fyd0m9l9.pdf

Message-passing algorithms for compressed sensing

Many real-world problems deal with collections of high-dimensional data, such as images, videos, text, and web documents, DNA microarray data, and more. Often, such high-dimensional data lie close to low-dimensional structures corresponding to several classes or categories to which the data belong. In this paper, we propose and study an algorithm, called sparse subspace clustering, to cluster data points that lie in a union of low-dimensional subspaces. The key idea is that, among the infinitely many possible representations of a data point in terms of other points, a sparse representation corresponds to selecting a few points from the same subspace. This motivates solving a sparse optimization program whose solution is used in a spectral clustering framework to infer the clustering of the data into subspaces. Since solving the sparse optimization program is in general NP-hard, we consider a convex relaxation and show that, under appropriate conditions on the arrangement of the subspaces and the distribution of the data, the proposed minimization program succeeds in recovering the desired sparse representations. The proposed algorithm is efficient and can handle data points near the intersections of subspaces. Another key advantage of the proposed algorithm with respect to the state of the art is that it can deal directly with data nuisances, such as noise, sparse outlying entries, and missing entries, by incorporating the model of the data into the sparse optimization program. We demonstrate the effectiveness of the proposed algorithm through experiments on synthetic data as well as the two real-world problems of motion segmentation and face clustering.

/pdf/sparse-subspace-clustering-algorithm-theory-and-applications-3fjsiby0gs.pdf

Sparse Subspace Clustering: Algorithm, Theory, and Applications

The basis pursuit problem seeks a minimum one-norm solution of an underdetermined least-squares problem. Basis pursuit denoise (BPDN) fits the least-squares problem only approximately, and a single parameter determines a curve that traces the optimal trade-off between the least-squares fit and the one-norm of the solution. We prove that this curve is convex and continuously differentiable over all points of interest, and show that it gives an explicit relationship to two other optimization problems closely related to BPDN. We describe a root-finding algorithm for finding arbitrary points on this curve; the algorithm is suitable for problems that are large scale and for those that are in the complex domain. At each iteration, a spectral gradient-projection method approximately minimizes a least-squares problem with an explicit one-norm constraint. Only matrix-vector operations are required. The primal-dual solution of this problem gives function and derivative information needed for the root-finding method. Numerical experiments on a comprehensive set of test problems demonstrate that the method scales well to large problems.

http://www.optimization-online.org/DB_FILE/2008/01/1889.pdf

Probing the Pareto Frontier for Basis Pursuit Solutions

We introduce a new estimator for the vector of coefficients β in the linear model y = Xβ + z, where X has dimensions n × p with p possibly larger than n. SLOPE, short for Sorted L-One Penalized Estimation, is the solution to [Formula: see text]where λ1 ≥ λ2 ≥ … ≥ λ p ≥ 0 and [Formula: see text] are the decreasing absolute values of the entries of b. This is a convex program and we demonstrate a solution algorithm whose computational complexity is roughly comparable to that of classical l1 procedures such as the Lasso. Here, the regularizer is a sorted l1 norm, which penalizes the regression coefficients according to their rank: the higher the rank-that is, stronger the signal-the larger the penalty. This is similar to the Benjamini and Hochberg [J. Roy. Statist. Soc. Ser. B57 (1995) 289-300] procedure (BH) which compares more significant p-values with more stringent thresholds. One notable choice of the sequence {λ i } is given by the BH critical values [Formula: see text], where q ∈ (0, 1) and z(α) is the quantile of a standard normal distribution. SLOPE aims to provide finite sample guarantees on the selected model; of special interest is the false discovery rate (FDR), defined as the expected proportion of irrelevant regressors among all selected predictors. Under orthogonal designs, SLOPE with λBH provably controls FDR at level q. Moreover, it also appears to have appreciable inferential properties under more general designs X while having substantial power, as demonstrated in a series of experiments running on both simulated and real data.

/pdf/slope-adaptive-variable-selection-via-convex-optimization-3rrtcnbvh6.pdf

SLOPE { Adaptive Variable Selection via Convex Optimization

The problem of recovering a matrix from an incomplete sampling of its entries—also known as matrix completion—arises in a wide variety of practical situations, including collaborative filtering, system identification, sensor localization, rank aggregation, and many more. While many of these applications have a relatively long history, recent advances in the closely related field of compressed sensing have enabled a burst of progress in the last few years, and we now have a strong base of theoretical results concerning matrix completion. A typical result from this literature is that a generic d × d matrix of rank r can be exactly recovered from O(r dpolylog(d)) randomly chosen entries. Similar results can be established in the case of noisy observations and approximately low-rank matrices. See [1] and references therein for further details. Although these results are quite impressive, there is an important gap between the statement of the problem as considered in the matrix completion literature and many of the most common applications discussed therein. As an example, consider collaborative filtering and the now-famous “Netflix problem.” In this setting, we assume that there is some unknown matrix whose entries each represent a rating for a particular user on a particular movie. Since any user will rate only a small subset of possible movies, we are only able to observe a small fraction of the total entries in the matrix, and our goal is to infer the unseen ratings from the observed ones. If the rating matrix is low-rank, then this would seem to be the exact problem studied in the matrix completion literature. However, there is a subtle difference: the theory developed in this literature generally assumes that observations consist of (possibly noisy) continuous-valued entries of the matrix, whereas in the Netflix problem the observations are “quantized” to the set of integers between 1 and 5. If we believe that it is possible for a user’s true rating for a particular movie to be, for example, 4.5, then we must account for the impact of this “quantization noise” on our recovery. Of course, one could potentially treat quantization simply as a form of bounded noise, but this is somewhat unsatisfying because the ratings aren’t just quantized — there are also hard limits placed on the minimum and maximum allowable ratings. (Why should we suppose that a movie given a rating of 5 could not have a true underlying rating of 6 or 7 or 10?) The inadequacy of standard matrix completion techniques in dealing with this effect is particularly pronounced when we consider recommender systems where each rating consists of a single bit representing a positive or negative rating (consider for example rating music on Pandora, the relevance of advertisements on Hulu, or posts on Reddit or MathOverflow). Similar situations arise in nearly every application that has been proposed for matrix completion, including the analysis of incomplete survey data, the recovery of pairwise distance matrices (multidimensional scaling), quantum state tomography, and many others. In such cases, the assumptions made in the existing theory of matrix completion do not apply, standard algorithms are ill-posed, and a new theory is required. In this work we describe the approach we take in [1] to extend the theory of matrix completion to the case of 1-bit observations. We consider a statistical model for such data where a binary output is generated according to a probability distribution which is parameterized by the corresponding entry of the unknown matrix M . The central question we ask is: “Given observations of this form, can we recover the underlying matrix?” Several new challenges arise when trying to develop a theory for 1-bit matrix completion. First, matrix completion is in some sense a more challenging problem than compressed sensing. Specifically, some additional difficulty arises because the set of low-rank matrices is “coherent” with single entry measurements—there will always be certain (sparse) low-rank matrices that we cannot hope to recover without essentially sampling every entry of the matrix. The typical way to deal with this possibility is to consider a reduced set of lowrank matrices by placing restrictions on the entry-wise maximum of the matrix or its singular vectors—informally, we require that the matrix is not too “spiky”. However, we introduce an entirely new dimension of ill-posedness by restricting ourselves to 1-bit observations. An example of this is described in detail in [1] and shows that in the case where we simply observe Y = sign(M), the problem of recovering M is illposed. To see this, let M = uv∗ for any vectors u,v ∈ R, and for simplicity assume that there are no zero entries in u or v. Now let ũ and ṽ be any vectors with the same sign pattern as u and v respectively. It is apparent that either M or M = ũṽ will yield the same observations Y , and thus M and M are indistinguishable. Note that while it is obvious that this 1-bit measurement process will destroy any information we have regarding the scaling of M , this ill-posedness remains even if we knew something about the scaling a priori (such as the Frobenius norm of M ). For any given set of observations, there will always be radically different possible matrices that are all consistent with observed measurements. After considering this example, the problem might seem hopeless. However, an interesting surprise is that when we add noise to the problem (that is, when we observe a subset of the matrix Y = sign(M + Z) where Z 6= 0 is an appropriate stochastic matrix) the picture completely changes—this noise has a “dithering” effect and the problem becomes well-posed. In fact, we will show that in this setting we can sometimes recover M to the same degree of accuracy that is possible when given access to completely unquantized measurements! In particular, under appropriate conditions, O(rd) measurements are sufficient to accurately recover M . We will provide an overview of these results and discuss a number of practical applications.

1-Bit matrix completion

An optimization algorithm for minimizing a smooth function over a convex set is described. Each iteration of the method computes a descent direction by minimizing, over the original constraints, a diagonal plus lowrank quadratic approximation to the function. The quadratic approximation is constructed using a limited-memory quasi-Newton update. The method is suitable for large-scale problems where evaluation of the function is substantially more expensive than projection onto the constraint set. Numerical experiments on one-norm regularized test problems indicate that the proposed method is competitive with state-of-the-art methods such as boundconstrained L-BFGS and orthant-wise descent. We further show that the method generalizes to a wide class of problems, and substantially improves on state-of-the-art methods for problems such as learning the structure of Gaussian graphical models and Markov random elds.

/pdf/optimizing-costly-functions-with-simple-constraints-a-2tbqcp6cax.pdf

Optimizing Costly Functions with Simple Constraints: A Limited-Memory Projected Quasi-Newton Algorithm

The joint-sparse recovery problem aims to recover, from sets of compressed measurements, unknown sparse matrices with nonzero entries restricted to a subset of rows. This is an extension of the single-measurement-vector (SMV) problem widely studied in compressed sensing. We study the recovery properties of two algorithms for problems with noiseless data and exact-sparse representation. First, we show that recovery using sum-of-norm minimization cannot exceed the uniform-recovery rate of sequential SMV using l 1 minimization, and that there are problems that can be solved with one approach, but not the other. Second, we study the performance of the ReMBo algorithm (M. Mishali and Y. Eldar, ?Reduce and boost: Recovering arbitrary sets of jointly sparse vectors,? IEEE Trans. Signal Process., vol. 56, no. 10, 4692-4702, Oct. 2008) in combination with l 1 minimization, and show how recovery improves as more measurements are taken. From this analysis, it follows that having more measurements than the number of linearly independent nonzero rows does not improve the potential theoretical recovery rate.

Ewout van den Berg

Papers

Probing the Pareto Frontier for Basis Pursuit Solutions

SLOPE { Adaptive Variable Selection via Convex Optimization

1-Bit matrix completion

Optimizing Costly Functions with Simple Constraints: A Limited-Memory Projected Quasi-Newton Algorithm

Theoretical and Empirical Results for Recovery From Multiple Measurements