Blake Woodworth

TL;DR: In this article, the authors studied implicit regularization when optimizing an underdetermined quadratic objective over a matrix $X$ with gradient descent on a factorization of X, and provided empirical and theoretical evidence that with small enough step sizes and initialization close enough to the origin, gradient descent converges to the minimum nuclear norm solution.

TL;DR: It is proved that (in the worst case) any algorithm requires at least $\epsilon^{-4}$ queries to find an stationary point, and establishes that stochastic gradient descent is minimax optimal in this model.

TL;DR: This work shows how the scale of the initialization controls the transition between the "kernel" and "rich" regimes and affects generalization properties in multilayer homogeneous models and highlights an interesting role for the width of a model in the case that the predictor is not identically zero at initialization.

TL;DR: It is conjecture and theoretical evidence that with small enough step sizes and initialization close enough to the origin, gradient descent on a full dimensional factorization converges to the minimum nuclear norm solution.

TL;DR: For quadratic objectives it is proved that local SGD strictly dominates minibatch SGD and that accelerated localSGD is minimax optimal for quadratics and for general convex objectives the first guarantee that at least sometimes improves over minibatches SGD is provided.