DVAE++: Discrete Variational Autoencoders with Overlapping Transformations

TLDR: The authors proposed a new class of smoothing transformations based on a mixture of two overlapping distributions, and showed that the proposed transformation can be used for training binary latent models with either directed or undirected priors.

Abstract: Training of discrete latent variable models remains challenging because passing gradient information through discrete units is difficult We propose a new class of smoothing transformations based on a mixture of two overlapping distributions, and show that the proposed transformation can be used for training binary latent models with either directed or undirected priors We derive a new variational bound to efficiently train with Boltzmann machine priors Using this bound, we develop DVAE++, a generative model with a global discrete prior and a hierarchy of convolutional continuous variables Experiments on several benchmarks show that overlapping transformations outperform other recent continuous relaxations of discrete latent variables including Gumbel-Softmax (Maddison et al, 2016; Jang et al, 2016), and discrete variational autoencoders (Rolfe 2016)