Rate of convergence

About: Rate of convergence is a research topic. Over the lifetime, 31257 publications have been published within this topic receiving 795334 citations. The topic is also known as: convergence rate.

signSGD: Compressed Optimisation for Non-Convex Problems

Abstract: Training large neural networks requires distributing learning across multiple workers, where the cost of communicating gradients can be a significant bottleneck signSGD alleviates this problem by transmitting just the sign of each minibatch stochastic gradient We prove that it can get the best of both worlds: compressed gradients and SGD-level convergence rate The relative $\ell_1/\ell_2$ geometry of gradients, noise and curvature informs whether signSGD or SGD is theoretically better suited to a particular problem On the practical side we find that the momentum counterpart of signSGD is able to match the accuracy and convergence speed of Adam on deep Imagenet models We extend our theory to the distributed setting, where the parameter server uses majority vote to aggregate gradient signs from each worker enabling 1-bit compression of worker-server communication in both directions Using a theorem by Gauss we prove that majority vote can achieve the same reduction in variance as full precision distributed SGD Thus, there is great promise for sign-based optimisation schemes to achieve fast communication and fast convergence Code to reproduce experiments is to be found at this https URL

Persistency of Excitation in Identification Using Radial Basis Function Approximants

Abstract: In this paper, identification algorithms whose convergence and rate of convergence hinge on the regressor vector being persistently exciting are discussed. It is then shown that if the regressor vector is constructed out of radial basis function approximants, it will be persistently exciting, provided a kind of "ergodic" condition is satisfied. In addition, bounds on parameters associated with the persistently exciting regressor vector are provided; these parameters are connected with both the convergence and rates of convergence of the algorithms involved.

The Sinkhorn-Knopp Algorithm: Convergence and Applications

Abstract: As long as a square nonnegative matrix $A$ contains sufficient nonzero elements, then the Sinkhorn-Knopp algorithm can be used to balance the matrix, that is, to find a diagonal scaling of $A$ that is doubly stochastic. It is known that the convergence is linear, and an upper bound has been given for the rate of convergence for positive matrices. In this paper we give an explicit expression for the rate of convergence for fully indecomposable matrices. We describe how balancing algorithms can be used to give a measure of web page significance. We compare the measure with some well known alternatives, including PageRank. We show that, with an appropriate modification, the Sinkhorn-Knopp algorithm is a natural candidate for computing the measure on enormous data sets.

Change-point estimation in ARCH models

Abstract: The paper studies the change-point problem and the cross-covariance function for ARCH models. Bounds for the cross-covariance function are derived and explicit formulae are obtained in special cases. Consistency of a cusum type change-point estimator is proved and its rate of convergence is established. A Hajek-Renyi type inequality is also proved. Results are obtained under weak moment assumptions.