Nicolas Loizou

TL;DR: Stochastic Gradient Push is studied, it is proved that SGP converges to a stationary point of smooth, non-convex objectives at the same sub-linear rate as SGD, and that all nodes achieve consensus.

TL;DR: In this article, a unified convergence analysis of decentralized SGD methods is presented for smooth SGD problems and the convergence rates interpolate between heterogeneous (non-identically distributed data) and iid-data settings, recovering linear convergence rates in many special cases, for instance for over-parametrized models.

TL;DR: This theorem describes the convergence of an infinite array of variants of SGD, each of which is associated with a specific probability law governing the data selection rule used to form mini-batches, and can determine the mini-batch size that optimizes the total complexity.

TL;DR: In this paper, the convergence of SGD under the arbitrary sampling paradigm is analyzed, and it is shown that the optimal mini-batch size is a function of the expected smoothness.

TL;DR: A novel concept, which is called stochastic momentum, aimed at decreasing the cost of performing the momentum step is proposed, and it is proved that in some sparse data regimes and for sufficiently small momentum parameters, these methods enjoy better overall complexity than methods with deterministic momentum.