We introduce a new and increasingly relevant setting for distributed optimization in machine learning, where the data defining the optimization are unevenly distributed over an extremely large number of nodes. The goal is to train a high-quality centralized model. We refer to this setting as Federated Optimization. In this setting, communication efficiency is of the utmost importance and minimizing the number of rounds of communication is the principal goal. 
A motivating example arises when we keep the training data locally on users' mobile devices instead of logging it to a data center for training. In federated optimziation, the devices are used as compute nodes performing computation on their local data in order to update a global model. We suppose that we have extremely large number of devices in the network --- as many as the number of users of a given service, each of which has only a tiny fraction of the total data available. In particular, we expect the number of data points available locally to be much smaller than the number of devices. Additionally, since different users generate data with different patterns, it is reasonable to assume that no device has a representative sample of the overall distribution. 
We show that existing algorithms are not suitable for this setting, and propose a new algorithm which shows encouraging experimental results for sparse convex problems. This work also sets a path for future research needed in the context of \federated optimization.

/pdf/federated-optimization-distributed-machine-learning-for-on-3mxdc9gfx1.pdf

Federated Optimization: Distributed Machine Learning for On-Device Intelligence

Coordinate descent algorithms solve optimization problems by successively performing approximate minimization along coordinate directions or coordinate hyperplanes. They have been used in applications for many years, and their popularity continues to grow because of their usefulness in data analysis, machine learning, and other areas of current interest. This paper describes the fundamentals of the coordinate descent approach, together with variants and extensions and their convergence properties, mostly with reference to convex objectives. We pay particular attention to a certain problem structure that arises frequently in machine learning applications, showing that efficient implementations of accelerated coordinate descent algorithms are possible for problems of this type. We also present some parallel variants and discuss their convergence properties under several models of parallel execution.

http://www.optimization-online.org/DB_FILE/2014/12/4679.pdf

Coordinate descent algorithms

Coordinate Descent Algorithms

In 1963, Polyak proposed a simple condition that is sufficient to show a global linear convergence rate for gradient descent. This condition is a special case of the Łojasiewicz inequality proposed in the same year, and it does not require strong convexity or even convexity. In this work, we show that this much-older Polyak-Łojasiewicz PL inequality is actually weaker than the main conditions that have been explored to show linear convergence rates without strong convexity over the last 25 years. We also use the PL inequality to give new analyses of coordinate descent and stochastic gradient for many non-strongly-convex and some non-convex functions. We further propose a generalization that applies to proximal-gradient methods for non-smooth optimization, leading to simple proofs of linear convergence for support vector machines and L1-regularized least squares without additional assumptions.

Linear Convergence of Gradient and Proximal-Gradient Methods Under the Polyak-Łojasiewicz Condition

We obtain an improved finite-sample guarantee on the linear convergence of stochastic gradient descent for smooth and strongly convex objectives, improving from a quadratic dependence on the conditioning $$(L/\mu )^2$$(L/μ)2 (where $$L$$L is a bound on the smoothness and $$\mu $$μ on the strong convexity) to a linear dependence on $$L/\mu $$L/μ. Furthermore, we show how reweighting the sampling distribution (i.e. importance sampling) is necessary in order to further improve convergence, and obtain a linear dependence in the average smoothness, dominating previous results. We also discuss importance sampling for SGD more broadly and show how it can improve convergence also in other scenarios. Our results are based on a connection we make between SGD and the randomized Kaczmarz algorithm, which allows us to transfer ideas between the separate bodies of literature studying each of the two methods. In particular, we recast the randomized Kaczmarz algorithm as an instance of SGD, and apply our results to prove its exponential convergence, but to the solution of a weighted least squares problem rather than the original least squares problem. We then present a modified Kaczmarz algorithm with partially biased sampling which does converge to the original least squares solution with the same exponential convergence rate.

https://proceedings.neurips.cc/paper/2014/file/f29c21d4897f78948b91f03172341b7b-Paper.pdf

Stochastic Gradient Descent, Weighted Sampling, and the Randomized Kaczmarz algorithm

We describe an asynchronous parallel stochastic coordinate descent algorithm for minimizing smooth unconstrained or separably constrained functions. The method achieves a linear convergence rate on functions that satisfy an essential strong convexity property and a sublinear rate (1/K) on general convex functions. Near-linear speedup on a multicore system can be expected if the number of processors is O(n1/2) in unconstrained optimization and O(n1/4) in the separable-constrained case, where n is the number of variables. We describe results from implementation on 40-core processors.

/pdf/an-asynchronous-parallel-stochastic-coordinate-descent-57ee5u46xq.pdf

An Asynchronous Parallel Stochastic Coordinate Descent Algorithm

We describe an asynchronous parallel stochastic coordinate descent algorithm for minimizing smooth unconstrained or separably constrained functions The method achieves a linear convergence rate on functions that satisfy an essential strong convexity property and a sublinear rate (1/K) on general convex functions Near-linear speedup on a multicore system can be expected if the number of processors is O(n1/2) in unconstrained optimization and O(n1/4) in the separable-constrained case, where n is the number of variables We describe results from implementation on 40-core processors

/pdf/an-asynchronous-parallel-stochastic-coordinate-descent-1o5dxh9qev.pdf

An asynchronous parallel stochastic coordinate descent algorithm

We describe an asynchronous parallel stochastic coordinate descent algorithm for minimizing smooth unconstrained or separably constrained functions. The method achieves a linear convergence rate on functions that satisfy an essential strong convexity property and a sublinear rate ($1/K$) on general convex functions. Near-linear speedup on a multicore system can be expected if the number of processors is $O(n^{1/2})$ in unconstrained optimization and $O(n^{1/4})$ in the separable-constrained case, where $n$ is the number of variables. We describe results from implementation on 40-core processors.

We describe an asynchronous parallel variant of the randomized Kaczmarz (RK) algorithm for solving the linear system Ax = b. The analysis shows linear convergence and indicates that nearly linear speedup can be expected if the number of processors is bounded by a multiple of the number of rows in A.

An Asynchronous Parallel Randomized Kaczmarz Algorithm

In this paper, we propose a load-based scheme for assigning channels to radio interfaces in multi-radio, multi-channel wireless mesh networks. We first construct a model for channel assignment as an optimization problem with the goal of minimizing the overall network interference. The problem is proven to be NP-Hard. We then apply the Lagrangian relaxation method to obtain lower bounds as well as near-optimal feasible solutions for large size networks. We further present a meta-heuristic based on genetic algorithms, which can yield good quality solutions for very large networks. With these two centralized approaches as the benchmark, we propose a fully distributed algorithm in order to tackle the channel assignment problem practically. Our extensive simulation experiments demonstrate that the distributed algorithm performs competitively and can serve as a practical and scalable solution to the channel assignment problem.

Srikrishna Sridhar

Papers

An Asynchronous Parallel Stochastic Coordinate Descent Algorithm

An asynchronous parallel stochastic coordinate descent algorithm

An Asynchronous Parallel Stochastic Coordinate Descent Algorithm

An Asynchronous Parallel Randomized Kaczmarz Algorithm

Channel assignment in multi-radio wireless mesh networks : A graph-theoretic approach