Block splitting for distributed optimization
Neal Parikh,Stephen Boyd +1 more
Reads0
Chats0
TLDR
A general purpose method for solving convex optimization problems in a distributed computing environment that allows for handling each sub-block of $$A$$A on a separate machine if the problem data includes a large linear operator or matrix, and is the only general Purpose method with this property.Abstract:
This paper describes a general purpose method for solving convex optimization problems in a distributed computing environment. In particular, if the problem data includes a large linear operator or matrix $$A$$
, the method allows for handling each sub-block of $$A$$
on a separate machine. The approach works as follows. First, we define a canonical problem form called graph form, in which we have two sets of variables related by a linear operator $$A$$
, such that the objective function is separable across these two sets of variables. Many types of problems are easily expressed in graph form, including cone programs and a wide variety of regularized loss minimization problems from statistics, like logistic regression, the support vector machine, and the lasso. Next, we describe graph projection splitting, a form of Douglas–Rachford splitting or the alternating direction method of multipliers, to solve graph form problems serially. Finally, we derive a distributed block splitting algorithm based on graph projection splitting. In a statistical or machine learning context, this allows for training models exactly with a huge number of both training examples and features, such that each processor handles only a subset of both. To the best of our knowledge, this is the only general purpose method with this property. We present several numerical experiments in both the serial and distributed settings.read more
Citations
More filters
Book
Proximal Algorithms
Neal Parikh,Stephen Boyd +1 more
TL;DR: The many different interpretations of proximal operators and algorithms are discussed, their connections to many other topics in optimization and applied mathematics are described, some popular algorithms are surveyed, and a large number of examples of proxiesimal operators that commonly arise in practice are provided.
Journal ArticleDOI
Data-Driven Distributionally Robust Optimization Using the Wasserstein Metric: Performance Guarantees and Tractable Reformulations
TL;DR: In this paper, the authors consider stochastic programs where the distribution of the uncertain parameters is only observable through a finite training dataset and use the Wasserstein metric to construct a ball in the space of probability distributions centered at the uniform distribution on the training samples.
Posted Content
Data-driven Distributionally Robust Optimization Using the Wasserstein Metric: Performance Guarantees and Tractable Reformulations
TL;DR: It is demonstrated that the distributionally robust optimization problems over Wasserstein balls can in fact be reformulated as finite convex programs—in many interesting cases even as tractable linear programs.
Journal ArticleDOI
Conic Optimization via Operator Splitting and Homogeneous Self-Dual Embedding
TL;DR: In this article, the alternating directions method of multipliers is used to solve the homogeneous self-dual embedding, an equivalent feasibility problem involving finding a nonzero point in the intersection of a subspace and a cone.
Journal ArticleDOI
Parallel Multi-Block ADMM with o(1 / k) Convergence
TL;DR: The classic ADMM can be extended to the N-block Jacobi fashion and preserve convergence in the following two cases: (i) matrices A_i and Ai are mutually near-orthogonal and have full column-rank, or (ii) proximal terms are added to theN subproblems (but without any assumption on matrices $$A_i$$Ai).
References
More filters
Journal ArticleDOI
Regression Shrinkage and Selection via the Lasso
TL;DR: A new method for estimation in linear models called the lasso, which minimizes the residual sum of squares subject to the sum of the absolute value of the coefficients being less than a constant, is proposed.
Book
Convex Optimization
Stephen Boyd,Lieven Vandenberghe +1 more
TL;DR: In this article, the focus is on recognizing convex optimization problems and then finding the most appropriate technique for solving them, and a comprehensive introduction to the subject is given. But the focus of this book is not on the optimization problem itself, but on the problem of finding the appropriate technique to solve it.
Book
Distributed Optimization and Statistical Learning Via the Alternating Direction Method of Multipliers
TL;DR: It is argued that the alternating direction method of multipliers is well suited to distributed convex optimization, and in particular to large-scale problems arising in statistics, machine learning, and related areas.
Journal ArticleDOI
Using SeDuMi 1.02, a MATLAB toolbox for optimization over symmetric cones
TL;DR: This paper describes how to work with SeDuMi, an add-on for MATLAB, which lets you solve optimization problems with linear, quadratic and semidefiniteness constraints by exploiting sparsity.
Journal ArticleDOI
Model selection and estimation in regression with grouped variables
Ming Yuan,Yi Lin +1 more
TL;DR: In this paper, instead of selecting factors by stepwise backward elimination, the authors focus on the accuracy of estimation and consider extensions of the lasso, the LARS algorithm and the non-negative garrotte for factor selection.