Many problems of recent interest in statistics and machine learning can be posed in the framework of convex optimization. Due to the explosion in size and complexity of modern datasets, it is increasingly important to be able to solve problems with a very large number of features or training examples. As a result, both the decentralized collection or storage of these datasets as well as accompanying distributed solution methods are either necessary or at least highly desirable. In this review, we argue that the alternating direction method of multipliers is well suited to distributed convex optimization, and in particular to large-scale problems arising in statistics, machine learning, and related areas. The method was developed in the 1970s, with roots in the 1950s, and is equivalent or closely related to many other algorithms, such as dual decomposition, the method of multipliers, Douglas–Rachford splitting, Spingarn's method of partial inverses, Dykstra's alternating projections, Bregman iterative algorithms for l1 problems, proximal methods, and others. After briefly surveying the theory and history of the algorithm, we discuss applications to a wide variety of statistical and machine learning problems of recent interest, including the lasso, sparse logistic regression, basis pursuit, covariance selection, support vector machines, and many others. We also discuss general distributed optimization, extensions to the nonconvex setting, and efficient implementation, including some details on distributed MPI and Hadoop MapReduce implementations.

/pdf/distributed-optimization-and-statistical-learning-via-the-8s9gtddw0d.pdf

Distributed Optimization and Statistical Learning Via the Alternating Direction Method of Multipliers

/pdf/convex-analysisnoer-san-nojin-zhan-nituite-5dl2rbmoyr.pdf

Convex Analysisの二,三の進展について

We consider a problem of considerable practical interest: the recovery of a data matrix from a sampling of its entries. Suppose that we observe m entries selected uniformly at random from a matrix M. Can we complete the matrix and recover the entries that we have not seen?

We show that one can perfectly recover most low-rank matrices from what appears to be an incomplete set of entries. We prove that if the number m of sampled entries obeys $$m\ge C\,n^{1.2}r\log n$$ for some positive numerical constant C, then with very high probability, most n×n matrices of rank r can be perfectly recovered by solving a simple convex optimization program. This program finds the matrix with minimum nuclear norm that fits the data. The condition above assumes that the rank is not too large. However, if one replaces the 1.2 exponent with 1.25, then the result holds for all values of the rank. Similar results hold for arbitrary rectangular matrices as well. Our results are connected with the recent literature on compressed sensing, and show that objects other than signals and images can be perfectly reconstructed from very limited information.

/pdf/exact-matrix-completion-via-convex-optimization-3nmwb4bacp.pdf

Exact Matrix Completion via Convex Optimization

In this paper we study a first-order primal-dual algorithm for non-smooth convex optimization problems with known saddle-point structure. We prove convergence to a saddle-point with rate O(1/N) in finite dimensions for the complete class of problems. We further show accelerations of the proposed algorithm to yield improved rates on problems with some degree of smoothness. In particular we show that we can achieve O(1/N 2) convergence on problems, where the primal or the dual objective is uniformly convex, and we can show linear convergence, i.e. O(? N ) for some ??(0,1), on smooth problems. The wide applicability of the proposed algorithm is demonstrated on several imaging problems such as image denoising, image deconvolution, image inpainting, motion estimation and multi-label image segmentation.

https://hal.archives-ouvertes.fr/hal-00490826/document

A First-Order Primal-Dual Algorithm for Convex Problems with Applications to Imaging

The formalism of probabilistic graphical models provides a unifying framework for capturing complex dependencies among random variables, and building large-scale multivariate statistical models. Graphical models have become a focus of research in many statistical, computational and mathematical fields, including bioinformatics, communication theory, statistical physics, combinatorial optimization, signal and image processing, information retrieval and statistical machine learning. Many problems that arise in specific instances — including the key problems of computing marginals and modes of probability distributions — are best studied in the general setting. Working with exponential family representations, and exploiting the conjugate duality between the cumulant function and the entropy for exponential families, we develop general variational representations of the problems of computing likelihoods, marginal probabilities and most probable configurations. We describe how a wide variety of algorithms — among them sum-product, cluster variational methods, expectation-propagation, mean field methods, max-product and linear programming relaxation, as well as conic programming relaxations — can all be understood in terms of exact or approximate forms of these variational representations. The variational approach provides a complementary alternative to Markov chain Monte Carlo as a general source of approximation methods for inference in large-scale statistical models.

/pdf/graphical-models-exponential-families-and-variational-3i6dwlfta1.pdf

Graphical Models, Exponential Families, and Variational Inference

We study a distributed computation model for optimizing a sum of convex objective functions corresponding to multiple agents. For solving this (not necessarily smooth) optimization problem, we consider a subgradient method that is distributed among the agents. The method involves every agent minimizing his/her own objective function while exchanging information locally with other agents in the network over a time-varying topology. We provide convergence results and convergence rate estimates for the subgradient method. Our convergence rate results explicitly characterize the tradeoff between a desired accuracy of the generated approximate optimal solutions and the number of iterations needed to achieve the accuracy.

http://www.ifp.illinois.edu/~angelia/distributed_journal_final.pdf

Distributed Subgradient Methods for Multi-Agent Optimization

Convex Analysis and Optimization

We present distributed algorithms that can be used by multiple agents to align their estimates with a particular value over a network with time-varying connectivity. Our framework is general in that this value can represent a consensus value among multiple agents or an optimal solution of an optimization problem, where the global objective function is a combination of local agent objective functions. Our main focus is on constrained problems where the estimates of each agent are restricted to lie in different convex sets. To highlight the effects of constraints, we first consider a constrained consensus problem and present a distributed "projected consensus algorithm" in which agents combine their local averaging operation with projection on their individual constraint sets. This algorithm can be viewed as a version of an alternating projection method with weights that are varying over time and across agents. We establish convergence and convergence rate results for the projected consensus algorithm. We next study a constrained optimization problem for optimizing the sum of local objective functions of the agents subject to the intersection of their local constraint sets. We present a distributed "projected subgradient algorithm" which involves each agent performing a local averaging operation, taking a subgradient step to minimize its own objective function, and projecting on its constraint set. We show that, with an appropriately selected stepsize rule, the agent estimates generated by this algorithm converge to the same optimal solution for the cases when the weights are constant and equal, and when the weights are time-varying but all agents have the same constraint set.

/pdf/constrained-consensus-and-optimization-in-multi-agent-4dept7vnxb.pdf

Constrained Consensus and Optimization in Multi-Agent Networks

We provide a framework for studying the relationship between the financial network architecture and the likelihood of systemic failures due to contagion of counterparty risk. We show that financial contagion exhibits a form of phase transition as interbank connections increase: as long as the magnitude and the number of negative shocks affecting financial institutions are sufficiently small, more "complete" interbank claims enhance the stability of the system. However, beyond a certain point, such interconnections start to serve as a mechanism for propagation of shocks and lead to a more fragile financial system. We also show that, under natural contracting assumptions, financial networks that emerge in equilibrium may be socially inefficient due to the presence of a network externality: even though banks take the effects of their lending, risk-taking and failure on their immediate creditors into account, they do not internalize the consequences of their actions on the rest of the network.

https://papers.ssrn.com/sol3/Delivery.cfm/SSRN_ID2500504_code1428438.pdf?abstractid=2207439&mirid=5&type=2

Systemic Risk and Stability in Financial Networks

This paper argues that in the presence of intersectoral input-output linkages, microeconomic idiosyncratic shocks may lead to aggregate fluctuations. In particular, it shows that, as the economy becomes more disaggregated, the rate at which aggregate volatility decays is determined by the structure of the network capturing such linkages. Our main results provide a characterization of this relationship in terms of the importance of different sectors as suppliers to their immediate customers as well as their role as indirect suppliers to chains of downstream sectors. Such higher-order interconnections capture the possibility of “cascade effects” whereby productivity shocks to a sector propagate not only to its immediate downstream customers, but also indirectly to the rest of the economy. Our results highlight that sizable aggregate volatility is obtained from sectoral idiosyncratic shocks only if there exists significant asymmetry in the roles that sectors play as suppliers to others, and that the “sparseness” of the input-output matrix is unrelated to the nature of aggregate fluctuations.

/pdf/the-network-origins-of-aggregate-fluctuations-3yoh56cin3.pdf

Asuman Ozdaglar

Papers

Distributed Subgradient Methods for Multi-Agent Optimization

Convex Analysis and Optimization

Constrained Consensus and Optimization in Multi-Agent Networks

Systemic Risk and Stability in Financial Networks

The Network Origins of Aggregate Fluctuations