We introduce a new algorithm named WGAN, an alternative to traditional GAN training. In this new model, we show that we can improve the stability of learning, get rid of problems like mode collapse, and provide meaningful learning curves useful for debugging and hyperparameter searches. Furthermore, we show that the corresponding optimization problem is sound, and provide extensive theoretical work highlighting the deep connections to different distances between distributions.

/pdf/wasserstein-generative-adversarial-networks-2hd65i7hnr.pdf

Wasserstein Generative Adversarial Networks

Couplings and changes of variables.- Three examples of coupling techniques.- The founding fathers of optimal transport.- Qualitative description of optimal transport.- Basic properties.- Cyclical monotonicity and Kantorovich duality.- The Wasserstein distances.- Displacement interpolation.- The Monge-Mather shortening principle.- Solution of the Monge problem I: global approach.- Solution of the Monge problem II: Local approach.- The Jacobian equation.- Smoothness.- Qualitative picture.- Optimal transport and Riemannian geometry.- Ricci curvature.- Otto calculus.- Displacement convexity I.- Displacement convexity II.- Volume control.- Density control and local regularity.- Infinitesimal displacement convexity.- Isoperimetric-type inequalities.- Concentration inequalities.- Gradient flows I.- Gradient flows II: Qualitative properties.- Gradient flows III: Functional inequalities.- Synthetic treatment of Ricci curvature.- Analytic and synthetic points of view.- Convergence of metric-measure spaces.- Stability of optimal transport.- Weak Ricci curvature bounds I: Definition and Stability.- Weak Ricci curvature bounds II: Geometric and analytic properties.

Optimal Transport: Old and New

Generative Adversarial Networks (GANs) are powerful generative models, but suffer from training instability. The recently proposed Wasserstein GAN (WGAN) makes progress toward stable training of GANs, but sometimes can still generate only low-quality samples or fail to converge. We find that these problems are often due to the use of weight clipping in WGAN to enforce a Lipschitz constraint on the critic, which can lead to undesired behavior. We propose an alternative to clipping weights: penalize the norm of gradient of the critic with respect to its input. Our proposed method performs better than standard WGAN and enables stable training of a wide variety of GAN architectures with almost no hyperparameter tuning, including 101-layer ResNets and language models over discrete data. We also achieve high quality generations on CIFAR-10 and LSUN bedrooms.

Improved Training of Wasserstein GANs

Notation.- Notation.- Gradient Flow in Metric Spaces.- Curves and Gradients in Metric Spaces.- Existence of Curves of Maximal Slope and their Variational Approximation.- Proofs of the Convergence Theorems.- Uniqueness, Generation of Contraction Semigroups, Error Estimates.- Gradient Flow in the Space of Probability Measures.- Preliminary Results on Measure Theory.- The Optimal Transportation Problem.- The Wasserstein Distance and its Behaviour along Geodesics.- Absolutely Continuous Curves in p(X) and the Continuity Equation.- Convex Functionals in p(X).- Metric Slope and Subdifferential Calculus in (X).- Gradient Flows and Curves of Maximal Slope in p(X).

Gradient Flows: In Metric Spaces and in the Space of Probability Measures

Optimal transport distances are a fundamental family of distances for probability measures and histograms of features. Despite their appealing theoretical properties, excellent performance in retrieval tasks and intuitive formulation, their computation involves the resolution of a linear program whose cost can quickly become prohibitive whenever the size of the support of these measures or the histograms' dimension exceeds a few hundred. We propose in this work a new family of optimal transport distances that look at transport problems from a maximum-entropy perspective. We smooth the classic optimal transport problem with an entropic regularization term, and show that the resulting optimum is also a distance which can be computed through Sinkhorn's matrix scaling algorithm at a speed that is several orders of magnitude faster than that of transport solvers. We also show that this regularized distance improves upon classic optimal transport distances on the MNIST classification problem.

/pdf/sinkhorn-distances-lightspeed-computation-of-optimal-5exlqebm7y.pdf

Sinkhorn Distances: Lightspeed Computation of Optimal Transport

Introduction The Kantorovich duality Geometry of optimal transportation Brenier's polar factorization theorem The Monge-Ampere equation Displacement interpolation and displacement convexity Geometric and Gaussian inequalities The metric side of optimal transportation A differential point of view on optimal transportation Entropy production and transportation inequalities Problems Bibliography Table of short statements Index.

Topics in Optimal Transportation

We dene a notion of a measured length space X having nonnegative N-Ricci curvature, for N 2 [1;1), or having1-Ricci curvature bounded below byK, forK2 R. The denitions are in terms of the displacement convexity of certain functions on the associated Wasserstein metric space P2(X) of probability measures. We show that these properties are preserved under measured Gromov-Hausdor limits. We give geometric and analytic consequences. This paper has dual goals. One goal is to extend results about optimal transport from the setting of smooth Riemannian manifolds to the setting of length spaces. A second goal is to use optimal transport to give a notion for a measured length space to have Ricci curvature bounded below. We refer to [11] and [44] for background material on length spaces and optimal transport, respectively. Further bibliographic notes on optimal transport are in Appendix F. In the present introduction we motivate the questions that we address and we state the main results. To start on the geometric side, there are various reasons to try to extend notions of curvature from smooth Riemannian manifolds to more general spaces. A fairly general setting is that of length spaces, meaning metric spaces (X;d) in which the distance between two points equals the inmum of the lengths of curves joining the points. In the rest of this introduction we assume that X is a compact length space. Alexandrov gave a good notion of a length space having \curvature bounded below by K", with K a real number, in terms of the geodesic triangles in X. In the case of a Riemannian manifold M with the induced length structure, one recovers the Riemannian notion of having sectional curvature bounded below by K. Length spaces with Alexandrov curvature bounded below by K behave nicely with respect to the GromovHausdor topology on compact metric spaces (modulo isometries); they form

/pdf/ricci-curvature-for-metric-measure-spaces-via-optimal-3t6zu9nl81.pdf

Ricci curvature for metric-measure spaces via optimal transport

https://wiki.duke.edu/download/attachments/9505471/ottovil.pdf?version=1

Cédric Villani

Papers

Optimal Transport: Old and New

Topics in Optimal Transportation

Ricci curvature for metric-measure spaces via optimal transport

Generalization of an Inequality by Talagrand and Links with the Logarithmic Sobolev Inequality

Chapter 2 – A Review of Mathematical Topics in Collisional Kinetic Theory