The paper proposes Metropolis adjusted Langevin and Hamiltonian Monte Carlo sampling methods defined on the Riemann manifold to resolve the shortcomings of existing Monte Carlo algorithms when sampling from target densities that may be high dimensional and exhibit strong correlations. The methods provide fully automated adaptation mechanisms that circumvent the costly pilot runs that are required to tune proposal densities for Metropolis-Hastings or indeed Hamiltonian Monte Carlo and Metropolis adjusted Langevin algorithms. This allows for highly efficient sampling even in very high dimensions where different scalings may be required for the transient and stationary phases of the Markov chain. The methodology proposed exploits the Riemann geometry of the parameter space of statistical models and thus automatically adapts to the local structure when simulating paths across this manifold, providing highly efficient convergence and exploration of the target density. The performance of these Riemann manifold Monte Carlo methods is rigorously assessed by performing inference on logistic regression models, log-Gaussian Cox point processes, stochastic volatility models and Bayesian estimation of dynamic systems described by non-linear differential equations. Substantial improvements in the time-normalized effective sample size are reported when compared with alternative sampling approaches. MATLAB code that is available from http://www.ucl.ac.uk/statistics/research/rmhmc allows replication of all the results reported.

Riemann manifold Langevin and Hamiltonian Monte Carlo methods

L E Reichl 1980 Austin: University of Texas Press xii + 709 pp price $29.95 I can thoroughly recommend this book and congratulate the author for having brought together so many aspects of statistical physics in a comprehensible manner. The subject matter is modern; all the recent developments such as stochastic theory, theories of quantum fluids and critical phenomena, nonlinear chemical physics and hydrodynamics are included and yet each topic is built up from its foundations.

A Modern Course in Statistical Physics

Information geometry is a new mathematical discipline which applies the methodology of differential geometry to statistics. Therefore, families of exponential distributions are considered as embedded manifolds, called statistical manifolds. This includes so important families like the multivariate normal or the gamma distributions. Fisher information — well known in information theory — becomes a metric on statistical manifolds. The Fisher information metric enables a hyperbolic structure on the multivariate normal distributions. Information geometry offers new methods for hypothesis testings, estimation theory or stochastic filtering. These can be used in engineering areas like signal processing or video processing or finance.

http://ieeexplore.ieee.org/iel5/6423297/6450609/06450636.pdf

Information geometry and its applications

This article addresses a timely problem, especially within the context of the Internet, which is accessible by anyone in any discipline from virtually anywhere in the world. The problem of cross-language retrieval is not new. Then at now, cross-language information retrieval was seen to be a function that would facilitate the effective search for, exchange of, and retrieval information. This chapter reviews research and practice in CLIR that allows users to state queries in their native language and retrieve documents in any other language supported by the system. CLIR can simplify searching by multilingual users and, if translation resources are limited, can allow monolingual users to allocate those resources to the more promising documents. This review begins with an examination of the literature on user needs for CLIR. The chapter largely follows the retrieval system model (document preprocessing, query formulation, matching, selection, and delivery). Each section highlights the unique requirements imposed on one more stages of the model in cross-language retrieval applications. The authors describe evaluation techniques and conclude with observations regarding future directions for CLIR research.

Cross-Language Information Retrieval.

The Variational Auto-Encoder (VAE) is one of the most used unsupervised machine learning models. But although the default choice of a Gaussian distribution for both the prior and posterior represents a mathematically convenient distribution often leading to competitive results, we show that this parameterization fails to model data with a latent hyperspherical structure. To address this issue we propose using a von Mises-Fisher (vMF) distribution instead, leading to a hyperspherical latent space. Through a series of experiments, we show how such a hyperspherical VAE, or S-VAE, is more suitable for capturing data with a hyperspherical latent structure, while outperforming a normal, N-VAE, in low dimensions on other data types.

/pdf/hyperspherical-variational-auto-encoders-2dd1k2qqwe.pdf

Hyperspherical variational auto-encoders

Current supervised learning can learn spurious correlation during the data-fitting process, imposing issues regarding interpretability, out-of-distribution (OOD) generalization, and robustness. To avoid spurious correlation, we propose a \textbf{La}tent \textbf{C}ausal \textbf{I}nvariance \textbf{M}odel (LaCIM) which pursues \emph{causal prediction}. Specifically, we introduce latent variables that are separated into (a) output-causative factors and (b) others that are spuriously correlated to the output via confounders, to model the underlying causal factors. We further assume the generating mechanisms from latent space to observed data to be \emph{causally invariant}. We give the identifiable claim of such invariance, particularly the disentanglement of output-causative factors from others, as a theoretical guarantee for precise inference and avoiding spurious correlation. We propose a Variational-Bayesian-based method for estimation and to optimize over the latent space for prediction. The utility of our approach is verified by improved interpretability, prediction power on various OOD scenarios (including healthcare) and robustness on security.

Latent Causal Invariant Model

We consider doing Bayesian inference by minimizing the KL divergence on the 2-Wasserstein space $\mathcal{P}_2$. By exploring the Riemannian structure of $\mathcal{P}_2$, we develop two inference methods by simulating the gradient flow on $\mathcal{P}_2$ via updating particles, and an acceleration method that speeds up all such particle-simulation-based inference methods. Moreover we analyze the approximation flexibility of such methods, and conceive a novel bandwidth selection method for the kernel that they use. We note that $\mathcal{P}_2$ is quite abstract and general so that our methods can make closer approximation, while it still has a rich structure that enables practical implementation. Experiments show the effectiveness of the two proposed methods and the improvement of convergence by the acceleration method.

Accelerated First-order Methods on the Wasserstein Space for Bayesian Inference.

The generative adversarial network (GAN) has received considerable attention recently as a model for data synthesis, without an explicit specification of a likelihood function. There has been commensurate interest in leveraging likelihood estimates to improve GAN training. To enrich the understanding of this fast-growing yet almost exclusively heuristic-driven subject, we elucidate the theoretical roots of some of the empirical attempts to stabilize and improve GAN training with the introduction of likelihoods. We highlight new insights from variational theory of diffusion processes to derive a likelihood-based regularizing scheme for GAN training, and present a novel approach to train GANs with an unnormalized distribution instead of empirical samples. To substantiate our claims, we provide experimental evidence on how our theoretically-inspired new algorithms improve upon current practice.

/pdf/variational-annealing-of-gans-a-langevin-perspective-28zfp4590e.pdf

Variational annealing of GANs: A Langevin perspective

Stickers with vivid and engaging expressions are becoming increasingly popular in online messaging apps, and some works are dedicated to automatically select sticker response by matching text labels of stickers with previous utterances. However, due to their large quantities, it is impractical to require text labels for the all stickers. Hence, in this paper, we propose to recommend an appropriate sticker to user based on multi-turn dialog context history without any external labels. Two main challenges are confronted in this task. One is to learn semantic meaning of stickers without corresponding text labels. Another challenge is to jointly model the candidate sticker with the multi-turn dialog context. To tackle these challenges, we propose a sticker response selector (SRS) model. Specifically, SRS first employs a convolutional based sticker image encoder and a self-attention based multi-turn dialog encoder to obtain the representation of stickers and utterances. Next, deep interaction network is proposed to conduct deep matching between the sticker with each utterance in the dialog history. SRS then learns the short-term and long-term dependency between all interaction results by a fusion network to output the the final matching score. To evaluate our proposed method, we collect a large-scale real-world dialog dataset with stickers from one of the most popular online chatting platform. Extensive experiments conducted on this dataset show that our model achieves the state-of-the-art performance for all commonly-used metrics. Experiments also verify the effectiveness of each component of SRS. To facilitate further research in sticker selection field, we release this dataset of 340K multi-turn dialog and sticker pairs1.

Learning to Respond with Stickers: A Framework of Unifying Multi-Modality in Multi-Turn Dialog

Molecular conformation generation aims to generate three-dimensional coordinates of all the atoms in a molecule and is an important task in bioinformatics and pharmacology. Previous methods usually first predict the interatomic distances, the gradients of interatomic distances or the local structures (e.g., torsion angles) of a molecule, and then reconstruct its 3D conformation. How to directly generate the conformation without the above intermediate values is not fully explored. In this work, we propose a method that directly predicts the coordinates of atoms: (1) the loss function is invariant to roto-translation of coordinates and permutation of symmetric atoms; (2) the newly proposed model adaptively aggregates the bond and atom information and iteratively refines the coordinates of the generated conformation. Our method achieves the best results on GEOM-QM9 and GEOM-Drugs datasets. Further analysis shows that our generated conformations have closer properties (e.g., HOMO-LUMO gap) with the groundtruth conformations. In addition, our method improves molecular docking by providing better initial conformations. All the results demonstrate the effectiveness of our method and the great potential of the direct approach. The code is released at https://github.com/DirectMolecularConfGen/DMCG

Chang Liu

Papers

Latent Causal Invariant Model

Accelerated First-order Methods on the Wasserstein Space for Bayesian Inference.

Variational annealing of GANs: A Langevin perspective

Learning to Respond with Stickers: A Framework of Unifying Multi-Modality in Multi-Turn Dialog

Direct Molecular Conformation Generation