Many problems of recent interest in statistics and machine learning can be posed in the framework of convex optimization. Due to the explosion in size and complexity of modern datasets, it is increasingly important to be able to solve problems with a very large number of features or training examples. As a result, both the decentralized collection or storage of these datasets as well as accompanying distributed solution methods are either necessary or at least highly desirable. In this review, we argue that the alternating direction method of multipliers is well suited to distributed convex optimization, and in particular to large-scale problems arising in statistics, machine learning, and related areas. The method was developed in the 1970s, with roots in the 1950s, and is equivalent or closely related to many other algorithms, such as dual decomposition, the method of multipliers, Douglas–Rachford splitting, Spingarn's method of partial inverses, Dykstra's alternating projections, Bregman iterative algorithms for l1 problems, proximal methods, and others. After briefly surveying the theory and history of the algorithm, we discuss applications to a wide variety of statistical and machine learning problems of recent interest, including the lasso, sparse logistic regression, basis pursuit, covariance selection, support vector machines, and many others. We also discuss general distributed optimization, extensions to the nonconvex setting, and efficient implementation, including some details on distributed MPI and Hadoop MapReduce implementations.

/pdf/distributed-optimization-and-statistical-learning-via-the-8s9gtddw0d.pdf

Distributed Optimization and Statistical Learning Via the Alternating Direction Method of Multipliers

We develop fast algorithms for estimation of generalized linear models with convex penalties. The models include linear regression, two-class logistic regression, and multinomial regression problems while the penalties include l(1) (the lasso), l(2) (ridge regression) and mixtures of the two (the elastic net). The algorithms use cyclical coordinate descent, computed along a regularization path. The methods can handle large problems and can also deal efficiently with sparse features. In comparative timings we find that the new algorithms are considerably faster than competing methods.

/pdf/regularization-paths-for-generalized-linear-models-via-39r1t606ac.pdf

Regularization Paths for Generalized Linear Models via Coordinate Descent

The sparsity which is implicit in MR images is exploited to significantly undersample k -space. Some MR images such as angiograms are already sparse in the pixel representation; other, more complicated images have a sparse representation in some transform domain–for example, in terms of spatial finite-differences or their wavelet coefficients. According to the recently developed mathematical theory of compressedsensing, images with a sparse representation can be recovered from randomly undersampled k -space data, provided an appropriate nonlinear recovery scheme is used. Intuitively, artifacts due to random undersampling add as noise-like interference. In the sparse transform domain the significant coefficients stand out above the interference. A nonlinear thresholding scheme can recover the sparse coefficients, effectively recovering the image itself. In this article, practical incoherent undersampling schemes are developed and analyzed by means of their aliasing interference. Incoherence is introduced by pseudo-random variable-density undersampling of phase-encodes. The reconstruction is performed by minimizing the 1 norm of a transformed image, subject to data

/pdf/sparse-mri-the-application-of-compressed-sensing-for-rapid-3533njds12.pdf

Sparse MRI: The application of compressed sensing for rapid MR imaging.

It is now well understood that (1) it is possible to reconstruct sparse signals exactly from what appear to be highly incomplete sets of linear measurements and (2) that this can be done by constrained l1 minimization. In this paper, we study a novel method for sparse signal recovery that in many situations outperforms l1 minimization in the sense that substantially fewer measurements are needed for exact recovery. The algorithm consists of solving a sequence of weighted l1-minimization problems where the weights used for the next iteration are computed from the value of the current solution. We present a series of experiments demonstrating the remarkable performance and broad applicability of this algorithm in the areas of sparse signal recovery, statistical estimation, error correction and image processing. Interestingly, superior gains are also achieved when our method is applied to recover signals with assumed near-sparsity in overcomplete representations—not by reweighting the l1 norm of the coefficient sequence as is common, but by reweighting the l1 norm of the transformed object. An immediate consequence is the possibility of highly efficient data acquisition protocols by improving on a technique known as Compressive Sensing.

/pdf/enhancing-sparsity-by-reweighted-l-1-minimization-znwoz4koxr.pdf

Enhancing Sparsity by Reweighted ℓ 1 Minimization

CoSaMP: Iterative signal recovery from incomplete and inaccurate samples

Logistic regression with l1 regularization has been proposed as a promising method for feature selection in classification problems Several specialized solution methods have been proposed for l1-regularized logistic regression problems (LRPs) However, existing methods do not scale well to large problems that arise in many practical settings In this paper we describe an efficient interior-point method for solving l1-regularized LRPS Small problems with up to a thousand or so features and examples can be solved in seconds on a PC A variation on the basic method, that uses a preconditioned conjugate gradient method to compute the search step, can solve large sparse problems, with a million features and examples (eg, the 20 Newsgroups data set), in a few tens of minutes, on a PC Numerical experiments show that our method outperforms standard methods for solving convex optimization problems as well as other methods specifically designed for l1- regularized LRPs

A method for large-scale l 1 -regularized logistic regression

Convex loss minimization with lscr1 regularization has been proposed as a promising method for feature selection in classification (e.g., lscr1-regularized logistic regression) and regression (e.g., lscr1-regularized least squares). In this paper we describe an efficient interior-point method for solving large-scale lscr1-regularized convex loss minimization problems that uses a preconditioned conjugate gradient method to compute the search step. The method can solve very large problems. For example, the method can solve an lscr1-regularized logistic regression problem with a million features and examples (e.g., the 20 Newsgroups data set), in a few minutes, on a PC.

/pdf/an-efficient-method-for-large-scale-l-1-regularized-convex-1bphsaryte.pdf

An Efficient Method for Large-Scale l 1 -Regularized Convex Loss Minimization

We propose a sequential method to estimate monotone convex functions that consists of: (i) monotone regression via solving a constrained least square (LS) problem and (ii) convexification of the monotone regression estimate via solving a uniform approximation problem with associated constraints. We show that this method is faster than the constrained LS method. The ratio of computation time increases as data size increases. Moreover, we show that, under an appropriate smoothness condition, the uniform convergence rate achieved by the proposed method is nearly comparable to the best achievable rate for a non-parametric estimate which ignores the shape constraint. Simulation studies show that our method is comparable to the constrained LS method in estimation error. We illustrate our method by analysing ground water level data of wells in Korea.

Estimating monotone convex functions via sequential shape modification

Contingency tables are often used to display the multivariate frequency distribution of variables of interest. Under the common multinomial assumption, the first step of contingency table analysis is to estimate cell probabilities. It is well known that the unconstrained maximum likelihood estimator (MLE) is given by cell counts divided by the total number of observations. However, in the presence of (complex) constraints on the unknown cell probabilities or their functions, the MLE or other types of estimators may often have no closed form and have to be obtained numerically. In this paper, we focus on finding the MLE of cell probabilities in contingency tables under two common types of constraints: known marginals and ordered marginals/conditionals, and propose a novel approach based on geometric programming. We present two important applications that illustrate the usefulness of our approach via comparison with existing methods. Further, we show that our GP-based approach is flexible, readily implementable, effort-saving and can provide a unified framework for various types of constrained estimation of cell probabilities in contingency tables.

Estimating cell probabilities in contingency tables with constraints on marginals/conditionals by geometric programming with applications

We consider maximum likelihood estimation (MLE) of heteroscedastic regression models based on a new “parametrization” of the likelihood in terms of the Sharpe ratio function, or the ratio of the mean and volatility functions. While with a standard parametrization the MLE problem is not convex and hence hard to solve globally, our parametrization leads to a functional that is jointly convex in the Sharpe ratio and inverse volatility functions. The major difficulty with the resulting infinite-dimensional convex program is the shape constraint on the inverse volatility function. We propose to solve the problem by solving a sequence of finite-dimensional convex programs with increasing dimensions, which can be done globally and efficiently. We demonstrate that, when the goal is to estimate the Sharpe ratio function directly, the finite-sample performance of the proposed estimation method is superior to existing methods that estimate the mean and variance functions separately. When applied to a financial dataset, our method captures a well-known covariate-dependent effect on the Shape ratio.

/pdf/nonparametric-sharpe-ratio-function-estimation-in-4fa0u96kx5.pdf

Seung-Jean Kim

Papers

A method for large-scale l 1 -regularized logistic regression

An Efficient Method for Large-Scale l 1 -Regularized Convex Loss Minimization

Estimating monotone convex functions via sequential shape modification

Estimating cell probabilities in contingency tables with constraints on marginals/conditionals by geometric programming with applications

Nonparametric Sharpe Ratio Function Estimation in Heteroscedastic Regression Models via Convex Optimization