Convergence of Probability Measures. By P. Billingsley. Chichester, Sussex, Wiley, 1968. xii, 253 p. 9 1/4“. 117s.

Convergence of Probability Measures

“Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks”の学習報告

We explore in this chapter questions of existence and uniqueness for solutions to stochastic differential equations and offer a study of their properties. This endeavor is really a study of diffusion processes. Loosely speaking, the term diffusion is attributed to a Markov process which has continuous sample paths and can be characterized in terms of its infinitesimal generator.

Stochastic Differential Equations

IEEE Transactions on Pattern Analysis and Machine Intelligence

Preface to the revised edition Remarks on notation 1. Basic examples 2. Characterization and existence 3. Stable processes and their extensions 4. The Levy-Ito decomposition of sample functions 5. Distributional properties of Levy processes 6. Subordination and density transformation 7. Recurrence and transience 8. Potential theory for Levy processes 9. Wiener-Hopf factorizations 10. More distributional properties Supplement Solutions to exercises References and author index Subject index.

Lévy processes and infinitely divisible distributions

The Wasserstein distance and its variations, e.g., the sliced-Wasserstein (SW) distance, have recently drawn attention from the machine learning community. The SW distance, specifically, was shown to have similar properties to the Wasserstein distance, while being much simpler to compute, and is therefore used in various applications including generative modeling and general supervised/unsupervised learning. In this paper, we first clarify the mathematical connection between the SW distance and the Radon transform. We then utilize the generalized Radon transform to define a new family of distances for probability measures, which we call generalized sliced-Wasserstein (GSW) distances. We further show that, similar to the SW distance, the GSW distance can be extended to a maximum GSW (max-GSW) distance. We then provide the conditions under which GSW and max-GSW distances are indeed proper metrics. Finally, we compare the numerical performance of the proposed distances on the generative modeling task of SW flows and report favorable results.

/pdf/generalized-sliced-wasserstein-distances-3ngd27sry9.pdf

Generalized Sliced Wasserstein Distances

We derive algorithms for generalised tensor factorisation (GTF) by building upon the well-established theory of Generalised Linear Models. Our algorithms are general in the sense that we can compute arbitrary factorisations in a message passing framework, derived for a broad class of exponential family distributions including special cases such as Tweedie's distributions corresponding to β-divergences. By bounding the step size of the Fisher Scoring iteration of the GLM, we obtain general updates for real data and multiplicative updates for non-negative data. The GTF framework is, then extended easily to address the problems when multiple observed tensors are factorised simultaneously. We illustrate our coupled factorisation approach on synthetic data as well as on a musical audio restoration problem.

/pdf/generalised-coupled-tensor-factorisation-3jhhcfikpb.pdf

Generalised Coupled Tensor Factorisation

The gradient noise (GN) in the stochastic gradient descent (SGD) algorithm is often considered to be Gaussian in the large data regime by assuming that the classical central limit theorem (CLT) kicks in. This assumption is often made for mathematical convenience, since it enables SGD to be analyzed as a stochastic differential equation (SDE) driven by a Brownian motion. We argue that the Gaussianity assumption might fail to hold in deep learning settings and hence render the Brownian motion-based analyses inappropriate. Inspired by non-Gaussian natural phenomena, we consider the GN in a more general context and invoke the generalized CLT (GCLT), which suggests that the GN converges to a heavy-tailed $\alpha$-stable random variable. Accordingly, we propose to analyze SGD as an SDE driven by a Levy motion. Such SDEs can incur `jumps', which force the SDE transition from narrow minima to wider minima, as proven by existing metastability theory. To validate the $\alpha$-stable assumption, we conduct extensive experiments on common deep learning architectures and show that in all settings, the GN is highly non-Gaussian and admits heavy-tails. We further investigate the tail behavior in varying network architectures and sizes, loss functions, and datasets. Our results open up a different perspective and shed more light on the belief that SGD prefers wide minima.

A Tail-Index Analysis of Stochastic Gradient Noise in Deep Neural Networks

By building upon the recent theory that estab- lished the connection between implicit generative modeling (IGM) and optimal transport, in this study, we propose a novel parameter-free algo- rithm for learning the underlying distributions of complicated datasets and sampling from them. The proposed algorithm is based on a functional optimization problem, which aims at finding a measure that is close to the data distribution as much as possible and also expressive enough for generative modeling purposes. We formulate the problem as a gradient flow in the space of proba- bility measures. The connections between gradi- ent flows and stochastic differential equations let us develop a computationally efficient algorithm for solving the optimization problem. We provide formal theoretical analysis where we prove finite- time error guarantees for the proposed algorithm. To the best of our knowledge, the proposed algo- rithm is the first nonparametric IGM algorithm with explicit theoretical guarantees. Our experi- mental results support our theory and show that our algorithm is able to successfully capture the structure of different types of data distributions.

/pdf/sliced-wasserstein-flows-nonparametric-generative-modeling-3wgcbxcml3.pdf

Sliced-Wasserstein Flows: Nonparametric Generative Modeling via Optimal Transport and Diffusions

Interactive musical systems require real-time, low-latency, accurate, and reliable event detection and classification algorithms. In this paper, we introduce a model-based algorithm for detection of percussive events and test the algorithm on the detection and classification of different percussive sounds. We focus on tuning the algorithm for a good compromise between temporal precision, classification accuracy and low latency. The model is trained offline on different percussive sounds using the expectation maximization approach for learning spectral templates for each sound and is able to run online to detect and classify sounds from audio stream input by a Hidden Markov Model. Our results indicate that the approach is promising and applicable in design and development of interactive musical systems.

https://www.cmpe.boun.edu.tr/~cemgil/papers/simsekli-jylha-erkut-cemgil2011.pdf

Umut Simsekli

Papers

Generalized Sliced Wasserstein Distances

Generalised Coupled Tensor Factorisation

A Tail-Index Analysis of Stochastic Gradient Noise in Deep Neural Networks

Sliced-Wasserstein Flows: Nonparametric Generative Modeling via Optimal Transport and Diffusions

Real-time recognition of percussive sounds by a model-based method