Optimal transport distances are a fundamental family of distances for probability measures and histograms of features. Despite their appealing theoretical properties, excellent performance in retrieval tasks and intuitive formulation, their computation involves the resolution of a linear program whose cost can quickly become prohibitive whenever the size of the support of these measures or the histograms' dimension exceeds a few hundred. We propose in this work a new family of optimal transport distances that look at transport problems from a maximum-entropy perspective. We smooth the classic optimal transport problem with an entropic regularization term, and show that the resulting optimum is also a distance which can be computed through Sinkhorn's matrix scaling algorithm at a speed that is several orders of magnitude faster than that of transport solvers. We also show that this regularized distance improves upon classic optimal transport distances on the MNIST classification problem.

/pdf/sinkhorn-distances-lightspeed-computation-of-optimal-5exlqebm7y.pdf

Sinkhorn Distances: Lightspeed Computation of Optimal Transport

https://www.fc.up.pt/dmat/engmat/2013/seminario/apresentacoes/Marisa1.pdf

A new point matching algorithm for non-rigid registration

Some geometric properties of PD's are established, Kullback's $I$-divergence playing the role of squared Euclidean distance. The minimum discrimination information problem is viewed as that of projecting a PD onto a convex set of PD's and useful existence theorems for and characterizations of the minimizing PD are arrived at. A natural generalization of known iterative algorithms converging to the minimizing PD in special situations is given; even for those special cases, our convergence proof is more generally valid than those previously published. As corollaries of independent interest, generalizations of known results on the existence of PD's or nonnegative matrices of a certain form are obtained. The Lagrange multiplier technique is not used.

/pdf/i-divergence-geometry-of-probability-distributions-and-3ii060af2t.pdf

$I$-Divergence Geometry of Probability Distributions and Minimization Problems

Optimal transport (OT) theory can be informally described using the words of the French mathematician Gaspard Monge (1746-1818): A worker with a shovel in hand has to move a large pile of sand lying on a construction site. The goal of the worker is to erect with all that sand a target pile with a prescribed shape (for example, that of a giant sand castle). Naturally, the worker wishes to minimize her total effort, quantified for instance as the total distance or time spent carrying shovelfuls of sand. Mathematicians interested in OT cast that problem as that of comparing two probability distributions, two different piles of sand of the same volume. They consider all of the many possible ways to morph, transport or reshape the first pile into the second, and associate a "global" cost to every such transport, using the "local" consideration of how much it costs to move a grain of sand from one place to another. Recent years have witnessed the spread of OT in several fields, thanks to the emergence of approximate solvers that can scale to sizes and dimensions that are relevant to data sciences. Thanks to this newfound scalability, OT is being increasingly used to unlock various problems in imaging sciences (such as color or texture processing), computer vision and graphics (for shape manipulation) or machine learning (for regression, classification and density fitting). This short book reviews OT with a bias toward numerical methods and their applications in data sciences, and sheds lights on the theoretical properties of OT that make it particularly useful for some of these applications.

Computational Optimal Transport

The concave-convex procedure (CCCP) is a way to construct discrete-time iterative dynamical systems that are guaranteed to decrease global optimization and energy functions monotonically. This procedure can be applied to almost any optimization problem, and many existing algorithms can be interpreted in terms of it. In particular, we prove that all expectation-maximization algorithms and classes of Legendre minimization and variational bounding algorithms can be reexpressed in terms of CCCP. We show that many existing neural network and mean-field theory algorithms are also examples of CCCP. The generalized iterative scaling algorithm and Sinkhorn's algorithm can also be expressed as CCCP by changing variables. CCCP can be used both as a new way to understand, and prove the convergence of, existing optimization algorithms and as a procedure for generating new algorithms.

/pdf/the-concave-convex-procedure-2pz8vktufq.pdf

The concave-convex procedure

/pdf/a-relationship-between-arbitrary-positive-matrices-and-eqco9zw27d.pdf

A Relationship Between Arbitrary Positive Matrices and Doubly Stochastic Matrices

This paper is concerned with the condition for the convergence to a doubly stochastic limit of a sequence of matrices obtained from a nonnegative matrix A by alternately scaling the rows and columns of A and with the condition for the existence of diagonal matrices A and D2 with positive main diagonals such that ΏγAΏ2 is doubly stochastic. The result is the following. The sequence of matrices converges to a doubly stochastic limit if and only if the matrix A contains at least one positive diagonal. A necessary and sufficient condition that there exist diagonal matrices A and D2 with positive main diagonals such that D1AD2 is both doubly stochastic and the limit of the iteration is that AφO and each positive entry of A is contained in a positive diagonal. The form DιAD2 is unique, and A and D2 are unique up to a positive scalar multiple if and only if A is fully indecomposable.

/pdf/concerning-nonnegative-matrices-and-doubly-stochastic-1vxztftxdb.pdf

Concerning nonnegative matrices and doubly stochastic matrices

Diagonal Equivalence to Matrices with Prescribed Row and Column Sums

Let A be a nonnegative m x n matrix and let r= (rl, ** I rm) and c = (c1' * c ) be positive vectors such that ?m r. = zn. c1. It is well known that if there exists a nonnegative z =1 z ]-=1 m x n matrix B with the same zero pattern as A having the ith row sum ri and jth column sum c,, there exist diagonal matrices D1 and D with positive main diagonals such that D1AD2 has ith row sum r. and jth column sum cjHowever the known proofs are at best cumbersome. It is shown here that this result can be obtained by considering the minimum of a certain real-valued function of n positive variables. It has been shown originally by Sinkhorn and Knopp [81 and Brualdi, Parter, and Schneider [31 that if A is a nonnegative fully indecomposable matrix, i.e. A contains no s x (n s) zero submatrix, then there exists a doubly stochastic matrix of the form D 1AD2 where DI and D2 are diagonal matrices with positive main diagonals. Later Djokovic? [41, and independently, London [5], proved the same theorem by considering the minimum of

/pdf/diagonal-equivalence-to-matrices-with-prescribed-row-and-5g18mdvxpw.pdf

Diagonal equivalence to matrices with prescribed row and column sums. II

Nonnegative matrices characterization by diagonal products to imply properties of stochastic matrices

/pdf/problems-involving-diagonal-products-in-nonnegative-matrices-3vut17dhwy.pdf

Richard Sinkhorn

Papers

A Relationship Between Arbitrary Positive Matrices and Doubly Stochastic Matrices

Concerning nonnegative matrices and doubly stochastic matrices

Diagonal Equivalence to Matrices with Prescribed Row and Column Sums

Diagonal equivalence to matrices with prescribed row and column sums. II

Problems involving diagonal products in nonnegative matrices