Showing papers on "Probabilistic latent semantic analysis published in 2016"

Latent Embeddings for Zero-Shot Classification

Abstract: We present a novel latent embedding model for learning a compatibility function between image and class embeddings, in the context of zero-shot classification. The proposed method augments the state-of-the-art bilinear compatibility model by incorporating latent variables. Instead of learning a single bilinear map, it learns a collection of maps with the selection, of which map to use, being a latent variable for the current image-class pair. We train the model with a ranking based objective function which penalizes incorrect rankings of the true class for a given image. We empirically demonstrate that our model improves the state-of-the-art for various class embeddings consistently on three challenging publicly available datasets for the zero-shot setting. Moreover, our method leads to visually highly interpretable results with clear clusters of different fine-grained object properties that correspond to different latent variable maps.

Applied Latent Class Analysis

Abstract: Thank you very much for downloading applied latent class analysis. Maybe you have knowledge that, people have look numerous times for their chosen novels like this applied latent class analysis, but end up in harmful downloads. Rather than reading a good book with a cup of tea in the afternoon, instead they are facing with some malicious virus inside their laptop. applied latent class analysis is available in our book collection an online access to it is set as public so you can download it instantly. Our digital library saves in multiple locations, allowing you to get the most less latency time to download any of our books like this one. Merely said, the applied latent class analysis is universally compatible with any devices to read.

Dirichlet-Derived Multiple Topic Scene Classification Model for High Spatial Resolution Remote Sensing Imagery

Abstract: Due to the complex arrangements of the ground objects in high spatial resolution (HSR) imagery scenes, HSR imagery scene classification is a challenging task, which is aimed at bridging the semantic gap between the low-level features and the high-level semantic concepts. A combination of multiple complementary features for HSR imagery scene classification is considered a potential way to improve the performance. However, the different types of features have different characteristics, and how to fuse the different types of features is a classic problem. In this paper, a Dirichlet-derived multiple topic model (DMTM) is proposed to fuse heterogeneous features at a topic level for HSR imagery scene classification. An efficient algorithm based on a variational expectation–maximization framework is developed to infer the DMTM and estimate the parameters of the DMTM. The proposed DMTM scene classification method is able to incorporate different types of features with different characteristics, no matter whether these features are local or global, discrete or continuous. Meanwhile, the proposed DMTM can also reduce the dimension of the features representing the HSR images. In our experiments, three types of heterogeneous features, i.e., the local spectral feature, the local structural feature, and the global textural feature, were employed. The experimental results with three different HSR imagery data sets show that the three types of features are complementary. In addition, the proposed DMTM is able to reduce the dimension of the features representing the HSR images, to fuse the different types of features efficiently, and to improve the performance of the scene classification over that of other scene classification algorithms based on spatial pyramid matching, probabilistic latent semantic analysis, and latent Dirichlet allocation.

TopicRNN: A Recurrent Neural Network with Long-Range Semantic Dependency

Abstract: In this paper, we propose TopicRNN, a recurrent neural network (RNN)-based language model designed to directly capture the global semantic meaning relating words in a document via latent topics. Because of their sequential nature, RNNs are good at capturing the local structure of a word sequence – both semantic and syntactic – but might face difficulty remembering long-range dependencies. Intuitively, these long-range dependencies are of semantic nature. In contrast, latent topic models are able to capture the global underlying semantic structure of a document but do not account for word ordering. The proposed TopicRNN model integrates the merits of RNNs and latent topic models: it captures local (syntactic) dependencies using an RNN and global (semantic) dependencies using latent topics. Unlike previous work on contextual RNN language modeling, our model is learned end-to-end. Empirical results on word prediction show that TopicRNN outperforms existing contextual RNN baselines. In addition, TopicRNN can be used as an unsupervised feature extractor for documents. We do this for sentiment analysis and report a new state-of-the-art error rate on the IMDB movie review dataset that amounts to a 13.3% improvement over the previous best result. Finally TopicRNN also yields sensible topics, making it a useful alternative to document models such as latent Dirichlet allocation.

Variational inference for latent variables and uncertain inputs in Gaussian processes

Abstract: The Gaussian process latent variable model (GP-LVM) provides a flexible approach for non-linear dimensionality reduction that has been widely applied. However, the current approach for training GP-LVMs is based on maximum likelihood, where the latent projection variables are maximised over rather than integrated out. In this paper we present a Bayesian method for training GP-LVMs by introducing a non-standard variational inference framework that allows to approximately integrate out the latent variables and subsequently train a GP-LVM by maximising an analytic lower bound on the exact marginal likelihood. We apply this method for learning a GP-LVM from i.i.d. observations and for learning non-linear dynamical systems where the observations are temporally correlated. We show that a benefit of the variational Bayesian procedure is its robustness to overfitting and its ability to automatically select the dimensionality of the non-linear latent space. The resulting framework is generic, flexible and easy to extend for other purposes, such as Gaussian process regression with uncertain or partially missing inputs. We demonstrate our method on synthetic data and standard machine learning benchmarks, as well as challenging real world datasets, including high resolution video data.

Understanding urban mobility patterns with a probabilistic tensor factorization framework

Abstract: The rapid developments of ubiquitous mobile computing provide planners and researchers with new opportunities to understand and build smart cities by mining the massive spatial-temporal mobility data. However, given the increasing complexity and volume of the emerging mobility datasets, it also becomes challenging to build novel analytical framework that is capable of understanding the structural properties and critical features. In this paper, we introduce an analytical framework to deal with high-dimensional human mobility data. To this end, we formulate mobility data in a probabilistic setting and consider each record a multivariate observation sampled from an underlying distribution. In order to characterize this distribution, we use a multi-way probabilistic factorization model based on the concept of tensor decomposition and probabilistic latent semantic analysis (PLSA). The model provides us with a flexible approach to understand multi-way mobility involving higher-order interactions—which are difficult to characterize with conventional approaches—using simple latent structures. The model can be efficiently estimated using the expectation maximization (EM) algorithm. As a numerical example, this model is applied on a four-way dataset recording 14 million public transport journeys extracted from smart card transactions in Singapore. This framework can shed light on the modeling of urban structure by understanding mobility flows in both spatial and temporal dimensions.

SpectralAnalysis: Software for the Masses

Abstract: The amount of data produced by spectral imaging techniques, such as mass spectrometry imaging, is rapidly increasing as technology and instrumentation advances. This, combined with an increasingly multimodal approach to analytical science, presents a significant challenge in the handling of large data from multiple sources. Here, we present software that can be used through the entire analysis workflow, from raw data through preprocessing (including a wide range of methods for smoothing, baseline correction, normalization, and image generation) to multivariate analysis (for example, memory efficient principal component analysis (PCA), non-negative matrix factorization (NMF), maximum autocorrelation factor (MAF), and probabilistic latent semantic analysis (PLSA)), for data sets acquired from single experiments to large multi-instrument, multimodality, and multicenter studies. SpectralAnalysis was also developed with extensibility in mind to stimulate development, comparisons, and evaluation of data analysi...

Learning Latent Vector Spaces for Product Search

Abstract: We introduce a novel latent vector space model that jointly learns the latent representations of words, e-commerce products and a mapping between the two without the need for explicit annotations. The power of the model lies in its ability to directly model the discriminative relation between products and a particular word. We compare our method to existing latent vector space models (LSI, LDA and word2vec) and evaluate it as a feature in a learning to rank setting. Our latent vector space model achieves its enhanced performance as it learns better product representations. Furthermore, the mapping from words to products and the representations of words benefit directly from the errors propagated back from the product representations during parameter estimation. We provide an in-depth analysis of the performance of our model and analyze the structure of the learned representations.

PixelVAE: A Latent Variable Model for Natural Images

Abstract: Natural image modeling is a landmark challenge of unsupervised learning. Variational Autoencoders (VAEs) learn a useful latent representation and model global structure well but have difficulty capturing small details. PixelCNN models details very well, but lacks a latent code and is difficult to scale for capturing large structures. We present PixelVAE, a VAE model with an autoregressive decoder based on PixelCNN. Our model requires very few expensive autoregressive layers compared to PixelCNN and learns latent codes that are more compressed than a standard VAE while still capturing most non-trivial structure. Finally, we extend our model to a hierarchy of latent variables at different scales. Our model achieves state-of-the-art performance on binarized MNIST, competitive performance on 64x64 ImageNet, and high-quality samples on the LSUN bedrooms dataset.

Joint Modelling of Multiple Network Views

Abstract: Latent space models (LSM) for network data rely on the basic assumption that each node of the network has an unknown position in a D-dimensional Euclidean latent space: generally the smaller the distance between two nodes in the latent space, the greater their probability of being connected. In this article, we propose a variational inference approach to estimate the intractable posterior of the LSM. In many cases, different network views on the same set of nodes are available. It can therefore be useful to build a model able to jointly summarize the information given by all the network views. For this purpose, we introduce the latent space joint model (LSJM) that merges the information given by multiple network views assuming that the probability of a node being connected with other nodes in each network view is explained by a unique latent variable. This model is demonstrated on the analysis of two datasets: an excerpt of 50 girls from “Teenage Friends and Lifestyle Study” data at three time points and ...

A Novel Approach to Extracting Non-Negative Latent Factors From Non-Negative Big Sparse Matrices

Abstract: An inherently non-negative latent factor model is proposed to extract non-negative latent factors from non-negative big sparse matrices efficiently and effectively. A single-element-dependent sigmoid function connects output latent factors with decision variables, such that non-negativity constraints on the output latent factors are always fulfilled and thus successfully separated from the training process with respect to the decision variables. Consequently, the proposed model can be easily and fast built with excellent prediction accuracy. Experimental results on an industrial size sparse matrix are given to verify its outstanding performance and suitability for industrial applications.

Organizing Books and Authors by Multilayer SOM

Abstract: This paper introduces a new framework for the organization of electronic books (e-books) and their corresponding authors using a multilayer self-organizing map (MLSOM). An author is modeled by a rich tree-structured representation, and an MLSOM-based system is used as an efficient solution to the organizational problem of structured data. The tree-structured representation formulates author features in a hierarchy of author biography, books, pages, and paragraphs. To efficiently tackle the tree-structured representation, we used an MLSOM algorithm that serves as a clustering technique to handle e-books and their corresponding authors. A book and author recommender system is then implemented using the proposed framework. The effectiveness of our approach was examined in a large-scale data set containing 3868 authors along with the 10500 e-books that they wrote. We also provided visualization results of MLSOM for revealing the relevance patterns hidden from presented author clusters. The experimental results corroborate that the proposed method outperforms other content-based models (e.g., rate adapting poisson, latent Dirichlet allocation, probabilistic latent semantic indexing, and so on) and offers a promising solution to book recommendation, author recommendation, and visualization.

Latent semantic analysis cosines as a cognitive similarity measure: Evidence from priming studies.

Abstract: In distributional semantics models (DSMs) such as latent semantic analysis (LSA), words are represented as vectors in a high-dimensional vector space. This allows for computing word similarities as the cosine of the angle between two such vectors. In two experiments, we investigated whether LSA cosine similarities predict priming effects, in that higher cosine similarities are associated with shorter reaction times (RTs). Critically, we applied a pseudo-random procedure in generating the item material to ensure that we directly manipulated LSA cosines as an independent variable. We employed two lexical priming experiments with lexical decision tasks (LDTs). In Experiment 1 we presented participants with 200 different prime words, each paired with one unique target. We found a significant effect of cosine similarities on RTs. The same was true for Experiment 2, where we reversed the prime-target order (primes of Experiment 1 were targets in Experiment 2, and vice versa). The results of these experiments confirm that LSA cosine similarities can predict priming effects, supporting the view that they are psychologically relevant. The present study thereby provides evidence for qualifying LSA cosine similarities not only as a linguistic measure, but also as a cognitive similarity measure. However, it is also shown that other DSMs can outperform LSA as a predictor of priming effects.

Properties of latent variable network models

Abstract: We derive properties of latent variable models for networks, a broad class of models that includes the widely used latent position models. We characterize several features of interest, with particular focus on the degree distribution, clustering coefficient, average path length, and degree correlations. We introduce the Gaussian latent position model, and derive analytic expressions and asymptotic approximations for its network properties. We pay particular attention to one special case, the Gaussian latent position model with random effects, and show that it can represent the heavy-tailed degree distributions, positive asymptotic clustering coefficients, and small-world behaviors that often occur in observed social networks. Finally, we illustrate the ability of the models to capture important features of real networks through several well-known datasets.

Aspect-based latent factor model by integrating ratings and reviews for recommender system

Abstract: Recommender system has been recognized as a superior way for solving personal information overload problem. Rating, as an evaluation criteria revealing how much a customer likes a product, has been a foundation of recommender systems for a long period based on the popular latent factor models. However, review texts as the valuable user generated content have been neglected all the time. Recently, models integrating ratings and review texts as training sources have attracted a lot of attention, which may model review texts by topic model or its variants and then link latent factor vectors to topic distribution of review texts. For that, the integrated models need complicated optimization algorithms to fuse the heterogeneous sources, that may cause greater errors.In this work, we aim to propose a novel model, called Aspect-based Latent Factor Model (ALFM) to integrate ratings and review texts via latent factor model, in which by integrating rating matrix, user-review matrix and item-attribute matrix, the user latent factors and item latent factors with word latent factors can be derived. Our proposed model aggregates all review texts of the same user on the respective items and builds a user-review matrix by word frequencies. Similarly, an item's review is considered as all review texts of the same item collected from respective users. According to different information abstracted from review texts, we introduce two different kinds of item-attribute matrix to integrate the item-word frequencies and polarity scores of corresponding words. Experimental results on real-world data sets from amazon.com illustrate that our model can not only perform better than traditional models and art-of-state models on rating prediction task, but also accomplish cross-domain task through transferring word embedding.

Cross-Modal Correlation Learning by Adaptive Hierarchical Semantic Aggregation

Abstract: With the explosive growth of web data, effective and efficient technologies are in urgent need for retrieving semantically relevant contents of heterogeneous modalities. Previous studies devote efforts to modeling simple cross-modal statistical dependencies, and globally projecting the heterogeneous modalities into a measurable subspace. However, global projections cannot appropriately adapt to diverse contents, and the naturally existing multilevel semantic relation in web data is ignored. We study the problem of semantic coherent retrieval, where documents from different modalities should be ranked by the semantic relevance to the query. Accordingly, we propose TINA, a correlation learning method by adaptive hierarchical semantic aggregation. First, by joint modeling of content and ontology similarities, we build a semantic hierarchy to measure multilevel semantic relevance. Second, with a set of local linear projections and probabilistic membership functions, we propose two paradigms for local expert aggregation, i.e., local projection aggregation and local distance aggregation. To learn the cross-modal projections, we optimize the structure risk objective function that involves semantic coherence measurement, local projection consistency, and the complexity penalty of local projections. Compared to existing approaches, a better bias-variance tradeoff is achieved by TINA in real-world cross-modal correlation learning tasks. Extensive experiments on widely used NUS-WIDE and ICML-Challenge for image–text retrieval demonstrate that TINA better adapts to the multilevel semantic relation and content divergence, and, thus, outperforms state of the art with better semantic coherence.

An Architecture for Deep, Hierarchical Generative Models

Abstract: We present an architecture which lets us train deep, directed generative models with many layers of latent variables. We include deterministic paths between all latent variables and the generated output, and provide a richer set of connections between computations for inference and generation, which enables more effective communication of information throughout the model during training. To improve performance on natural images, we incorporate a lightweight autoregressive model in the reconstruction distribution. These techniques permit end-to-end training of models with 10+ layers of latent variables. Experiments show that our approach achieves state-of-the-art performance on standard image modelling benchmarks, can expose latent class structure in the absence of label information, and can provide convincing imputations of occluded regions in natural images.

Joint Learning of Semantic and Latent Attributes

Abstract: As mid-level semantic properties shared across object categories, attributes have been studied extensively. Recent approaches have attempted joint modelling of multiple attributes together with class labels so as to exploit their correlations for better attribute prediction and object recognition. However, they often ignore the fact that there exist some shared properties other than nameable/semantic attributes, which we call latent attributes. Basically, they can be further divided into discriminative and non-discriminative parts depending on whether they can contribute to an object recognition task. We argue that learning the latent attributes jointly with user-defined semantic attributes not only leads to better representation for object recognition but also helps with semantic attribute prediction. A novel dictionary learning model is proposed which decomposes the dictionary space into three parts corresponding to semantic, latent discriminative and latent background attributes respectively. An efficient algorithm is then formulated to solve the resultant optimization problem. Extensive experiments show that the proposed attribute learning method produces state-of-the-art results on both attribute prediction and attribute-based person re-identification.

Ensemble multi-label text categorization based on rotation forest and latent semantic indexing

Abstract: Text categorization has gained increasing popularity in the last years due the explosive growth of multimedia documents. As a document can be associated with multiple non-exclusive categories simultaneously (e.g., Virus, Health, Sports, and Olympic Games), text categorization provides many opportunities for developing novel multi-label learning approaches devoted specifically to textual data. In this paper, we propose an ensemble multi-label classification method for text categorization based on four key ideas: (1) performing Latent Semantic Indexing based on distinct orthogonal projections on lower-dimensional spaces of concepts; (2) random splitting of the vocabulary; (3) document bootstrapping; and (4) the use of BoosTexter as a powerful multi-label base learner for text categorization to simultaneously encourage diversity and individual accuracy in the committee. Diversity of the ensemble is promoted through random splits of the vocabulary that leads to different orthogonal projections on lower-dimensional latent concept spaces. Accuracy of the committee members is promoted through the underlying latent semantic structure uncovered in the text. The combination of both rotation-based ensemble construction and Latent Semantic Indexing projection is shown to bring about significant improvements in terms of Average Precision, Coverage, Ranking loss and One error compared to five state-of-the-art approaches across 14 real-word textual data sets covering a wide variety of topics including health, education, business, science and arts.

Multi-scale blocks based image emotion classification using multiple instance learning

Abstract: Emotional factors usually affect users' preferences for and evaluations of images. Although affective image analysis attracts increasing attention, there are still three major challenges remaining: 1) it is difficult to classify an image into a single emotion type since different regions within an image can represent different emotions; 2) there is a gap between low-level features and high-level emotions and 3) it is difficult to collect a training set of reliable emotional image content. To address these three issues, we propose an emotion classification method based on multi-scale blocks using Multiple Instance Learning (MIL). We firstly extract blocks of an image at multiple scales using different image segmentation methods pyramid segmentation and simple linear iterative clustering (SLIC) and represent each block using the bag-of-visual-words (BoVW) method. Then, to bridge the “affective gap”, probabilistic latent semantic analysis (pLSA) is employed to estimate the latent topic distribution as a mid-level representation of each block. Finally, MIL, which reduces the need for exact labelling, is employed to classify the dominant emotion type of the image. Experiments carried out on three widely used datasets demonstrate that our proposed method with S-LIC effectively improves the state-of-the-art results of image emotion classification 5.1% on average.

Supervised Latent Factor Analysis for Process Data Regression Modeling and Soft Sensor Application

Abstract: This brief proposed a new supervised latent factor analysis (FA) method for process data regression modeling. Different from the traditional principal component analysis/regression model, the new model can successfully estimate heterogeneous variances from different process variables, which is more practical. Under the same probabilistic modeling framework, the single supervised latent FA model is further extended to the mixture form. Efficient expectation–maximization algorithms are developed for parameter learning in both single and mixture supervised latent FA models. Based on the regression modeling between easy-to-measure and difficult-to-measure process variables, two soft sensors are built for quality prediction in the process. Two case studies are provided to evaluate the modeling and performances of the new methods.

Robust semantic text similarity using LSA, machine learning, and linguistic resources

Abstract: Semantic textual similarity is a measure of the degree of semantic equivalence between two pieces of text. We describe the SemSim system and its performance in the *SEM 2013 and SemEval-2014 tasks on semantic textual similarity. At the core of our system lies a robust distributional word similarity component that combines latent semantic analysis and machine learning augmented with data from several linguistic resources. We used a simple term alignment algorithm to handle longer pieces of text. Additional wrappers and resources were used to handle task specific challenges that include processing Spanish text, comparing text sequences of different lengths, handling informal words and phrases, and matching words with sense definitions. In the *SEM 2013 task on Semantic Textual Similarity, our best performing system ranked first among the 89 submitted runs. In the SemEval-2014 task on Multilingual Semantic Textual Similarity, we ranked a close second in both the English and Spanish subtasks. In the SemEval-2014 task on Cross-Level Semantic Similarity, we ranked first in Sentence---Phrase, Phrase---Word, and Word---Sense subtasks and second in the Paragraph---Sentence subtask.

Cloud Extraction from Chinese High Resolution Satellite Imagery by Probabilistic Latent Semantic Analysis and Object-Based Machine Learning

Abstract: Automatic cloud extraction from satellite imagery is a vital process for many applications in optical remote sensing since clouds can locally obscure the surface features and alter the reflectance. Clouds can be easily distinguished by the human eyes in satellite imagery via remarkable regional characteristics, but finding a way to automatically detect various kinds of clouds by computer programs to speed up the processing efficiency remains a challenge. This paper introduces a new cloud detection method based on probabilistic latent semantic analysis (PLSA) and object-based machine learning. The method begins by segmenting satellite images into superpixels by Simple Linear Iterative Clustering (SLIC) algorithm while also extracting the spectral, texture, frequency and line segment features. Then, the implicit information in each superpixel is extracted from the feature histogram through the PLSA model by which the descriptor of each superpixel can be computed to form a feature vector for classification. Thereafter, the cloud mask is extracted by optimal thresholding and applying the Support Vector Machine (SVM) algorithm at the superpixel level. The GrabCut algorithm is then applied to extract more accurate cloud regions at the pixel level by assuming the cloud mask as the prior knowledge. When compared to different cloud detection methods in the literature, the overall accuracy of the proposed cloud detection method was up to 90 percent for ZY-3 and GF-1 images, which is about a 6.8 percent improvement over the traditional spectral-based methods. The experimental results show that the proposed method can automatically and accurately detect clouds using the multispectral information of the available four bands.

Micro-blog topic detection method based on BTM topic model and K-means clustering algorithm

Abstract: The development of micro-blog, generating large-scale short texts, provides people with convenient communication. In the meantime, discovering topics from short texts genuinely becomes an intractable problem. It was hard for traditional topic model-to-model short texts, such as probabilistic latent semantic analysis (PLSA) and Latent Dirichlet Allocation (LDA). They suffered from the severe data sparsity when disposed short texts. Moreover, K-means clustering algorithm can make topics discriminative when datasets is intensive and the difference among topic documents is distinct. In this paper, BTM topic model is employed to process short texts–micro-blog data for alleviating the problem of sparsity. At the same time, we integrating K-means clustering algorithm into BTM (Biterm Topic Model) for topics discovery further. The results of experiments on Sina micro-blog short text collections demonstrate that our method can discover topics effectively.

Bias-Adjusted Three-Step Latent Markov Modeling With Covariates

Abstract: Latent Markov models with covariates can be estimated via 1-step maximum likelihood. However, this 1-step approach has various disadvantages, such as that the inclusion of covariates in the model might alter the formation of the latent states and that parameter estimation could become infeasible with large numbers of time points, responses, and covariates. This is why researchers typically prefer performing the analysis in a stepwise manner; that is, they first construct the measurement model, then obtain the latent state classifications, and subsequently study the relationship between covariates and latent state memberships. However, such a stepwise approach yields downward-biased estimates of the covariate effects on initial state and transition probabilities. This article, shows how to overcome this problem using a generalization of the bias-corrected 3-step estimation method proposed for latent class analysis (Asparouhov & Muthen, 2014; Bolck, Croon, & Hagenaars, 2004; Vermunt, 2010). We give a formal...

A New multi-instance multi-label learning approach for image and text classification

Abstract: Recently, a reasonable and effectively framework to deal with the classification problem of the polysemy object with complex connotation is multi-instance multi-label (MIML) learning framework in which each example is not only represented by multiple instances but also associated with multiple labels. As we all know, feature expression plays an important role in the classification problems. It determines the accuracy of the classification results from the source. Considering its difficulties for automatically extracting the high-level features which are useful and noiseless for the MIML problem, so in this paper we present a general MIML framework by combining the feature learning technologies with machine learning technologies. Further, based on this framework, a new approach called CPNMIML which combines the probabilistic latent semantic analysis (PLSA) with the neural networks (NN) is proposed. In CPNMIML algorithm, we firstly learn the latent topic allocation of all the training examples by using the PLSA model, it is a feature learning process to get high-level features. Then we utilize the learned latent topic allocation of each training example to train the neural networks. Given a test example, we learn its latent topic distribution. Finally, we send the learned latent topic allocation of the test example to the trained neural networks to get the multiple labels of the test example. Experiments show that the proposed method has superior performance on two real-world MIML tasks.

Piecewise Latent Variables for Neural Variational Text Processing

Abstract: Advances in neural variational inference have facilitated the learning of powerful directed graphical models with continuous latent variables, such as variational autoencoders. The hope is that such models will learn to represent rich, multi-modal latent factors in real-world data, such as natural language text. However, current models often assume simplistic priors on the latent variables - such as the uni-modal Gaussian distribution - which are incapable of representing complex latent factors efficiently. To overcome this restriction, we propose the simple, but highly flexible, piecewise constant distribution. This distribution has the capacity to represent an exponential number of modes of a latent target distribution, while remaining mathematically tractable. Our results demonstrate that incorporating this new latent distribution into different models yields substantial improvements in natural language processing tasks such as document modeling and natural language generation for dialogue.