Many problems of recent interest in statistics and machine learning can be posed in the framework of convex optimization. Due to the explosion in size and complexity of modern datasets, it is increasingly important to be able to solve problems with a very large number of features or training examples. As a result, both the decentralized collection or storage of these datasets as well as accompanying distributed solution methods are either necessary or at least highly desirable. In this review, we argue that the alternating direction method of multipliers is well suited to distributed convex optimization, and in particular to large-scale problems arising in statistics, machine learning, and related areas. The method was developed in the 1970s, with roots in the 1950s, and is equivalent or closely related to many other algorithms, such as dual decomposition, the method of multipliers, Douglas–Rachford splitting, Spingarn's method of partial inverses, Dykstra's alternating projections, Bregman iterative algorithms for l1 problems, proximal methods, and others. After briefly surveying the theory and history of the algorithm, we discuss applications to a wide variety of statistical and machine learning problems of recent interest, including the lasso, sparse logistic regression, basis pursuit, covariance selection, support vector machines, and many others. We also discuss general distributed optimization, extensions to the nonconvex setting, and efficient implementation, including some details on distributed MPI and Hadoop MapReduce implementations.

/pdf/distributed-optimization-and-statistical-learning-via-the-8s9gtddw0d.pdf

Distributed Optimization and Statistical Learning Via the Alternating Direction Method of Multipliers

A novel and robust automated docking method that predicts the bound conformations of flexible ligands to macromolecular targets has been developed and tested, in combination with a new scoring function that estimates the free energy change upon binding. Interestingly, this method applies a Lamarckian model of genetics, in which environmental adaptations of an individual's phenotype are reverse transcribed into its genotype and become . heritable traits sic . We consider three search methods, Monte Carlo simulated annealing, a traditional genetic algorithm, and the Lamarckian genetic algorithm, and compare their performance in dockings of seven protein)ligand test systems having known three-dimensional structure. We show that both the traditional and Lamarckian genetic algorithms can handle ligands with more degrees of freedom than the simulated annealing method used in earlier versions of AUTODOCK, and that the Lamarckian genetic algorithm is the most efficient, reliable, and successful of the three. The empirical free energy function was calibrated using a set of 30 structurally known protein)ligand complexes with experimentally determined binding constants. Linear regression analysis of the observed binding constants in terms of a wide variety of structure-derived molecular properties was performed. The final model had a residual standard y1 y1 .

/pdf/automated-docking-using-a-lamarckian-genetic-algorithm-and-2t6y95r7fm.pdf

Automated docking using a Lamarckian genetic algorithm and an empirical binding free energy function

Statistics for Spatial Data.

/pdf/convex-analysisnoer-san-nojin-zhan-nituite-5dl2rbmoyr.pdf

Convex Analysisの二,三の進展について

Convergence of Probability Measures. By P. Billingsley. Chichester, Sussex, Wiley, 1968. xii, 253 p. 9 1/4“. 117s.

Convergence of Probability Measures

We give two general convergence proofs for random search algorithms. We review the literature and show how our results extend those available for specific variants of the conceptual algorithm studied here. We then exploit the convergence results to examine convergence rates and to actually design implementable methods. Finally we report on some computational experience.

/pdf/minimization-by-random-search-techniques-32qtk73kov.pdf

Minimization by Random Search Techniques

A common approach in coping with multiperiod optimization problems under uncertainty where statistical information is not really enough to support a stochastic programming model, has been to set up and analyze a number of scenarios. The aim then is to identify trends and essential features on which a robust decision policy can be based. This paper develops for the first time a rigorous algorithmic procedure for determining such a policy in response to any weighting of the scenarios. The scenarios are bundled at various levels to reflect the availability of information, and iterative adjustments are made to the decision policy to adapt to this structure and remove the dependence on hindsight.

/pdf/scenarios-and-policy-aggregation-in-optimization-under-5ibrtphkfy.pdf

Scenarios and policy aggregation in optimization under uncertainty

This paper gives an algorithm for L-shaped linear programs which arise naturally in optimal control problems with state constraints and stochastic linear programs (which can be represented in this form with an infinite number of linear constraints). The first section describes a cutting hyperplane algorithm which is shown to be equivalent to a partial decomposition algorithm of the dual program. The two last sections are devoted to applications of the cutting hyperplane algorithm to a linear optimal control problem and stochastic programming problems.

https://www.math.ucdavis.edu/~rjbw/mypage/Stochastic_Optimization_files/VnsW69_Lshape.pdf

L-shaped linear programs with applications to optimal control and stochastic programming.

We study the asymptotic behavior of the statistical estimators that maximize a not necessarily dieren tiable criterion function, possibly subject to side constraints (equalities and inequalities). The consistency results generalize those of Wald and Huber. Conditions are also given under which one is still able to obtain asymptotic normality. The analysis brings to the fore the relationship between the problem of nding statistical estimators and that of nding the optimal solutions of stochastic optimization problems with partial information. The last section is devoted to the properties of the saddle points of the associated Lagrangians.

/pdf/asymptotic-behavior-of-statistical-estimators-and-of-optimal-4b4kq52t2o.pdf

Asymptotic behavior of statistical estimators and of optimal solutions of stochastic optimization problems

To each stochastic program corresponds an equivalent deterministic program. The purpose of this paper is to compile and extend the known properties for the equivalent deterministic program of a sto...

Roger J.-B. Wets

Papers

Minimization by Random Search Techniques

Scenarios and policy aggregation in optimization under uncertainty

L-shaped linear programs with applications to optimal control and stochastic programming.

Asymptotic behavior of statistical estimators and of optimal solutions of stochastic optimization problems

Stochastic Programs with Fixed Recourse: The Equivalent Deterministic Program