Many problems of recent interest in statistics and machine learning can be posed in the framework of convex optimization. Due to the explosion in size and complexity of modern datasets, it is increasingly important to be able to solve problems with a very large number of features or training examples. As a result, both the decentralized collection or storage of these datasets as well as accompanying distributed solution methods are either necessary or at least highly desirable. In this review, we argue that the alternating direction method of multipliers is well suited to distributed convex optimization, and in particular to large-scale problems arising in statistics, machine learning, and related areas. The method was developed in the 1970s, with roots in the 1950s, and is equivalent or closely related to many other algorithms, such as dual decomposition, the method of multipliers, Douglas–Rachford splitting, Spingarn's method of partial inverses, Dykstra's alternating projections, Bregman iterative algorithms for l1 problems, proximal methods, and others. After briefly surveying the theory and history of the algorithm, we discuss applications to a wide variety of statistical and machine learning problems of recent interest, including the lasso, sparse logistic regression, basis pursuit, covariance selection, support vector machines, and many others. We also discuss general distributed optimization, extensions to the nonconvex setting, and efficient implementation, including some details on distributed MPI and Hadoop MapReduce implementations.

/pdf/distributed-optimization-and-statistical-learning-via-the-8s9gtddw0d.pdf

Distributed Optimization and Statistical Learning Via the Alternating Direction Method of Multipliers

Convergence of Probability Measures. By P. Billingsley. Chichester, Sussex, Wiley, 1968. xii, 253 p. 9 1/4“. 117s.

Convergence of Probability Measures

In this paper we study a first-order primal-dual algorithm for non-smooth convex optimization problems with known saddle-point structure. We prove convergence to a saddle-point with rate O(1/N) in finite dimensions for the complete class of problems. We further show accelerations of the proposed algorithm to yield improved rates on problems with some degree of smoothness. In particular we show that we can achieve O(1/N 2) convergence on problems, where the primal or the dual objective is uniformly convex, and we can show linear convergence, i.e. O(? N ) for some ??(0,1), on smooth problems. The wide applicability of the proposed algorithm is demonstrated on several imaging problems such as image denoising, image deconvolution, image inpainting, motion estimation and multi-label image segmentation.

https://hal.archives-ouvertes.fr/hal-00490826/document

A First-Order Primal-Dual Algorithm for Convex Problems with Applications to Imaging

This monograph is about a class of optimization algorithms called proximal algorithms. Much like Newton's method is a standard tool for solving unconstrained smooth optimization problems of modest size, proximal algorithms can be viewed as an analogous tool for nonsmooth, constrained, large-scale, or distributed versions of these problems. They are very generally applicable, but are especially well-suited to problems of substantial recent interest involving large or high-dimensional datasets. Proximal methods sit at a higher level of abstraction than classical algorithms like Newton's method: the base operation is evaluating the proximal operator of a function, which itself involves solving a small convex optimization problem. These subproblems, which generalize the problem of projecting a point onto a convex set, often admit closed-form solutions or can be solved very quickly with standard or simple specialized methods. Here, we discuss the many different interpretations of proximal operators and algorithms, describe their connections to many other topics in optimization and applied mathematics, survey some popular algorithms, and provide a large number of examples of proximal operators that commonly arise in practice.

Proximal Algorithms

Two convergence aspects of the EM algorithm are studied: (i) does the EM algorithm find a local maximum or a stationary value of the (incomplete-data) likelihood function? (ii) does the sequence of parameter estimates generated by EM converge? Several convergence results are obtained under conditions that are applicable to many practical situations Two useful special cases are: (a) if the unobserved complete-data specification can be described by a curved exponential family with compact parameter space, all the limit points of any EM sequence are stationary points of the likelihood function; (b) if the likelihood function is unimodal and a certain differentiability condition is satisfied, then any EM sequence converges to the unique maximum likelihood estimate A list of key properties of the algorithm is included

/pdf/on-the-convergence-properties-of-the-em-algorithm-zxjq0mqgzw.pdf

On the convergence properties of the em algorithm

Isotonic Regression. Tests of Ordered Hypotheses: The Normal Means Case. Tests of Ordered Hypotheses: Generalizations of the Likelihood Ratio Tests and Other Procedures. Inferences about a Set of Multinomial Parameters. Duality. Inferences Regarding Distributions Subject to Shape Restrictions. Conditional Expectation Given a sigma-lattice: Projections in a More General Setting. Complements. Tables. Bibliography and Related References. Index. Symbols and Acronyms.

Order restricted statistical inference

A commonly occurring problem in statistics is that of minimizing a least squares expression subject to side constraints. Here a simple iterative algorithm is presented and shown to converge to the desired solution. Several examples are presented, including finding the closest concave (convex) function to a set of points and other general quadratic programming problems. The dual problem to the basic problem is also discussed and a solution for it is given in terms of the algorithm. Finally, extensions to expressions other than least squares are given.

An Algorithm for Restricted Least Squares Regression

Many problems require the ability to find least squares projections onto convex regions. Here it is shown that if the constraint region can be expressed as a finite intersection of simpler convex regions, then one can obtain the projection onto the intersection by performing a series of projections only onto the simpler regions. This may facilitate finding least squares projections in more complicated spaces since the approach is valid for a general Hiibert space rather than just Rn.

A Method for Finding Projections onto the Intersection of Convex Sets in Hilbert Spaces

: It is suggested that problems in a reliability context may be handled by a Bayesian non-parametric approach. A stochastic process is defined whose sample paths may be assumed to be either increasing hazard rates or decreasing hazard rates by properly choosing the parameter functions of the process. The posterior distribution of the hazard rates are derived for both exact and censored data. Bayes estimates of hazard rates,c.d.f.'s, densities, and means, are found under squared error type loss functions. Some simulation is done and estimates graphed to better understand the estimators. Finally, estimates of the c.d.f. from some data in a paper by Kaplan and Meier are constructed. (Author)

/pdf/a-bayesian-nonparametric-approach-to-reliability-3vqqr0crbu.pdf

A Bayesian Nonparametric Approach to Reliability

https://www.kean.edu/~jwang/RTR15Reference/Dykstra_97.pdf

Richard L. Dykstra

Papers

Order restricted statistical inference

An Algorithm for Restricted Least Squares Regression

A Method for Finding Projections onto the Intersection of Convex Sets in Hilbert Spaces

A Bayesian Nonparametric Approach to Reliability

Stochastic comparisons of parallel systems of heterogeneous exponential components