Many problems of recent interest in statistics and machine learning can be posed in the framework of convex optimization. Due to the explosion in size and complexity of modern datasets, it is increasingly important to be able to solve problems with a very large number of features or training examples. As a result, both the decentralized collection or storage of these datasets as well as accompanying distributed solution methods are either necessary or at least highly desirable. In this review, we argue that the alternating direction method of multipliers is well suited to distributed convex optimization, and in particular to large-scale problems arising in statistics, machine learning, and related areas. The method was developed in the 1970s, with roots in the 1950s, and is equivalent or closely related to many other algorithms, such as dual decomposition, the method of multipliers, Douglas–Rachford splitting, Spingarn's method of partial inverses, Dykstra's alternating projections, Bregman iterative algorithms for l1 problems, proximal methods, and others. After briefly surveying the theory and history of the algorithm, we discuss applications to a wide variety of statistical and machine learning problems of recent interest, including the lasso, sparse logistic regression, basis pursuit, covariance selection, support vector machines, and many others. We also discuss general distributed optimization, extensions to the nonconvex setting, and efficient implementation, including some details on distributed MPI and Hadoop MapReduce implementations.

/pdf/distributed-optimization-and-statistical-learning-via-the-8s9gtddw0d.pdf

Distributed Optimization and Statistical Learning Via the Alternating Direction Method of Multipliers

We develop fast algorithms for estimation of generalized linear models with convex penalties. The models include linear regression, two-class logistic regression, and multinomial regression problems while the penalties include l(1) (the lasso), l(2) (ridge regression) and mixtures of the two (the elastic net). The algorithms use cyclical coordinate descent, computed along a regularization path. The methods can handle large problems and can also deal efficiently with sparse features. In comparative timings we find that the new algorithms are considerably faster than competing methods.

/pdf/regularization-paths-for-generalized-linear-models-via-39r1t606ac.pdf

Regularization Paths for Generalized Linear Models via Coordinate Descent

The sparsity which is implicit in MR images is exploited to significantly undersample k -space. Some MR images such as angiograms are already sparse in the pixel representation; other, more complicated images have a sparse representation in some transform domain–for example, in terms of spatial finite-differences or their wavelet coefficients. According to the recently developed mathematical theory of compressedsensing, images with a sparse representation can be recovered from randomly undersampled k -space data, provided an appropriate nonlinear recovery scheme is used. Intuitively, artifacts due to random undersampling add as noise-like interference. In the sparse transform domain the significant coefficients stand out above the interference. A nonlinear thresholding scheme can recover the sparse coefficients, effectively recovering the image itself. In this article, practical incoherent undersampling schemes are developed and analyzed by means of their aliasing interference. Incoherence is introduced by pseudo-random variable-density undersampling of phase-encodes. The reconstruction is performed by minimizing the 1 norm of a transformed image, subject to data

/pdf/sparse-mri-the-application-of-compressed-sensing-for-rapid-3533njds12.pdf

Sparse MRI: The application of compressed sensing for rapid MR imaging.

It is now well understood that (1) it is possible to reconstruct sparse signals exactly from what appear to be highly incomplete sets of linear measurements and (2) that this can be done by constrained l1 minimization. In this paper, we study a novel method for sparse signal recovery that in many situations outperforms l1 minimization in the sense that substantially fewer measurements are needed for exact recovery. The algorithm consists of solving a sequence of weighted l1-minimization problems where the weights used for the next iteration are computed from the value of the current solution. We present a series of experiments demonstrating the remarkable performance and broad applicability of this algorithm in the areas of sparse signal recovery, statistical estimation, error correction and image processing. Interestingly, superior gains are also achieved when our method is applied to recover signals with assumed near-sparsity in overcomplete representations—not by reweighting the l1 norm of the coefficient sequence as is common, but by reweighting the l1 norm of the transformed object. An immediate consequence is the possibility of highly efficient data acquisition protocols by improving on a technique known as Compressive Sensing.

/pdf/enhancing-sparsity-by-reweighted-l-1-minimization-znwoz4koxr.pdf

Enhancing Sparsity by Reweighted ℓ 1 Minimization

CoSaMP: Iterative signal recovery from incomplete and inaccurate samples

Many procedures have been proposed to compute nonparametric maximum likelihood estimators (NPMLEs) of survival functions under stochastic ordering constraints. However, each of them is only applicable to a specific type of stochastic ordering constraint and censoring, and is often hard to implement. In this article, we describe a general and flexible method based on geometric programming for computing the NPMLEs from right- or interval-censored data. To this end, we show that the monotonicity properties of the likelihood function and the stochastic ordering constraints considered in the literature allow us to reformulate the estimation problem as a geometric program (GP), a special type of mathematical optimization problem, which can be transformed to a convex optimization problem, and then solved globally and efficiently. We apply this GP-based method to real data examples to illustrate its generality in handling different types of ordering constraints and censoring. We also conduct simulation studies to...

Estimating Stochastically Ordered Survival Functions via Geometric Programming

A method for magnetic resonance imaging (MRI) is provided. A scanning path is specified. Gradient amplitude is determined as a function of arc-length along the scanning path in k-space. A time optimal gradient waveform for scanning the scanning path is calculated from the gradient amplitude. The scanning path is scanned using the time optimal gradient waveform.

MRI method of determining time-optimal gradient waveforms with gradient amplitude as a function of arc-length in k-space

Robust classification becomes challenging when each class consists of multiple subclasses. Examples include multi-font optical character recognition and automated protein function prediction. In correlation-based nearest-neighbor classification, the maximin correlation approach (MCA) provides the worst-case optimal solution by minimizing the maximum misclassification risk through an iterative procedure. Despite the optimality, the original MCA has drawbacks that have limited its wide applicability in practice. That is, the MCA tends to be sensitive to outliers, cannot effectively handle nonlinearities in datasets, and suffers from having high computational complexity. To address these limitations, we propose an improved solution, named regularized MCA (R-MCA). We first reformulate MCA as a quadratically constrained linear programming (QCLP) problem, incorporate regularization by introducing slack variables in the primal problem of the QCLP, and derive the corresponding Lagrangian dual. The dual formulation enables us to apply the kernel trick to R-MCA, so that it can better handle nonlinearities. Our experimental results demonstrate that the regularization and kernelization make the proposed R-MCA more robust and accurate for various classification tasks than the original MCA. Furthermore, when the data size or dimensionality grows, R-MCA runs substantially faster by solving either the primal or dual (whichever has a smaller variable dimension) of the QCLP. The source code of the proposed R-MCA methodology is available at http://data.snu.ac.kr/rmca .

Regularization and Kernelization of the Maximin Correlation Approach

An Interior-Point Method for Large-Scale {\it l}$_{\mbox{1}}$-Regularized Logistic Regression

In this paper we give a brief overview of a heuristic method for approximately solving a statistical digital circuit sizing problem, by reducing it to a related deterministic sizing problem that includes extra margins in each of the gate delays to account for the variation. Since the method is based on solving a deterministic sizing problem, it readily handles large-scale problems. Numerical experiments show that the resulting designs are often substantially better than one in which the variation in delay is ignored, and often quite close to the global optimum. Moreover, the designs seem to be good despite the simplicity of the statistical model (which ignores gate distribution shape, correlations, and so on). 
We illustrate the method on a 32-bit Ladner-Fischer adder, with a simple resistor-capacitor (RC) delay model, and a Pelgrom model of delay variation.

/pdf/a-heuristic-method-for-statistical-digital-circuit-sizing-3rjgiwtsmk.pdf

Seung-Jean Kim

Papers

Estimating Stochastically Ordered Survival Functions via Geometric Programming

MRI method of determining time-optimal gradient waveforms with gradient amplitude as a function of arc-length in k-space

Regularization and Kernelization of the Maximin Correlation Approach

An Interior-Point Method for Large-Scale {\it l}$_{\mbox{1}}$-Regularized Logistic Regression

A heuristic method for statistical digital circuit sizing