Generative Adversarial Networks (GANs) excel at creating realistic images with complex models for which maximum likelihood is infeasible. However, the convergence of GAN training has still not been proved. We propose a two time-scale update rule (TTUR) for training GANs with stochastic gradient descent on arbitrary GAN loss functions. TTUR has an individual learning rate for both the discriminator and the generator. Using the theory of stochastic approximation, we prove that the TTUR converges under mild assumptions to a stationary local Nash equilibrium. The convergence carries over to the popular Adam optimization, for which we prove that it follows the dynamics of a heavy ball with friction and thus prefers flat minima in the objective landscape. For the evaluation of the performance of GANs at image generation, we introduce the "Frechet Inception Distance" (FID) which captures the similarity of generated images to real ones better than the Inception Score. In experiments, TTUR improves learning for DCGANs and Improved Wasserstein GANs (WGAN-GP) outperforming conventional GAN training on CelebA, CIFAR-10, SVHN, LSUN Bedrooms, and the One Billion Word Benchmark.

GANs Trained by a Two Time-Scale Update Rule Converge to a Nash Equilibrium

We study the convergence of the proximal algorithm applied to nonsmooth functions that satisfy the Łjasiewicz inequality around their generalized critical points. Typical examples of functions complying with these conditions are continuous semialgebraic or subanalytic functions. Following Łjasiewicz’s original idea, we prove that any bounded sequence generated by the proximal algorithm converges to some generalized critical point. We also obtain convergence rate results which are related to the flatness of the function by means of Łjasiewicz exponents. Apart from the sharp and elliptic cases which yield finite or geometric convergence, the decay estimates that are derived are of the type O(k −s ), where s ∈ (0, + ∞) depends on the flatness of the function.

/pdf/on-the-convergence-of-the-proximal-algorithm-for-nonsmooth-2x1uc06qda.pdf

On the convergence of the proximal algorithm for nonsmooth functions involving analytic features

The ‘heavy ball with friction’ dynamical system x + γx + ∇f(x)=0 is a nonlinear oscillator with damping (γ>0). It has been recently proved that when H is a real Hilbert space and f: H→R is a differentiable convex function whose minimal value is achieved, then each solution trajectory t→x(t) of this system weakly converges towards a solution of ∇f(x)=0. We prove a similar result in the discrete setting for a general maximal monotone operator A by considering the following iterative method: x

k+1−x

 k
 −α
 k
 (x

 k
 −x

k−1)+λ
 k
 
A(x

k+1)∋0, giving conditions on the parameters λ
 k
 and α
 k
 in order to ensure weak convergence toward a solution of 0∈A(x) and extending classical convergence results concerning the standard proximal method.

An Inertial Proximal Method for Maximal Monotone Operators via Discretization of a Nonlinear Oscillator with Damping

We study the asymptotic behavior at infinity of solutions of a second order evolution equation with linear damping and convex potential. The differential system is defined in a real Hilbert space. It is proved that if the potential is bounded from below, then the solution trajectories are minimizing for it and converge weakly towards a minimizer of $\Phi$ if one exists; this convergence is strong when $\Phi$ is even or when the optimal set has a nonempty interior. We introduce a second order proximal-like iterative algorithm for the minimization of a convex function. It is defined by an implicit discretization of the continuous evolution problem and is valid for any closed proper convex function. We find conditions on some parameters of the algorithm in order to have a convergence result similar to the continuous case.

/pdf/on-the-minimizing-property-of-a-second-order-dissipative-3avucbw5iq.pdf

On the Minimizing Property of a Second Order Dissipative System in Hilbert Spaces

https://hal.univ-antilles.fr/hal-00780778/document

Convergence of a splitting inertial proximal method for monotone operators

Let H be a real Hilbert space and Φ:H ↦ R a continuously differentiable function, whose gradient is Lipschitz continuous on bounded sets. We study the nonlinear dissipative dynamical system: ${\ddot x}(t)+\lambda{\dot x}(t)+\nabla\Phi(x(t))=0, \lambda>0, plus Cauchy data, mainly in view of the unconstrained minimization of the function Φ. New results concerning the convergence of a solution to a critical point are given in various situations, including when Φ is convex (possibly with multiple minima) or is a Morse function (the critical point being then generically a local minimum); a counterexample shows that, without peculiar assumptions, a trajectory may not converge. By following the trajectories, we obtain a method for exploring local minima of Φ. A singular perturbation analysis links our results with those concerning gradient systems.

The heavy ball with friction method, i. the continuous dynamical system: global exploration of the local minima of a real-valued function by asymptotic analysis of a dissipative dynamical system

X. Goudou

Papers

The heavy ball with friction method, i. the continuous dynamical system: global exploration of the local minima of a real-valued function by asymptotic analysis of a dissipative dynamical system