Showing papers on "Latent variable model published in 2015"

A Recurrent Latent Variable Model for Sequential Data

Abstract: In this paper, we explore the inclusion of latent random variables into the dynamic hidden state of a recurrent neural network (RNN) by combining elements of the variational autoencoder. We argue that through the use of high-level latent random variables, the variational RNN (VRNN)1 can model the kind of variability observed in highly structured sequential data such as natural speech. We empirically evaluate the proposed model against related sequential models on four speech datasets and one handwriting dataset. Our results show the important roles that latent random variables can play in the RNN dynamic hidden state.

A recurrent latent variable model for sequential data

Abstract: In this paper, we explore the inclusion of latent random variables into the hidden state of a recurrent neural network (RNN) by combining the elements of the variational autoencoder. We argue that through the use of high-level latent random variables, the variational RNN (VRNN)1 can model the kind of variability observed in highly structured sequential data such as natural speech. We empirically evaluate the proposed model against other related sequential models on four speech datasets and one handwriting dataset. Our results show the important roles that latent random variables can play in the RNN dynamics.

Estimating and interpreting latent variable interactions: A tutorial for applying the latent moderated structural equations method

Abstract: Latent variables are common in psychological research. Research questions involving the interaction of two variables are likewise quite common. Methods for estimating and interpreting interactions between latent variables within a structural equation modeling framework have recently become available. The latent moderated structural equations (LMS) method is one that is built into Mplus software. The potential utility of this method is limited by the fact that the models do not produce traditional model fit indices, standardized coefficients, or effect sizes for the latent interaction, which renders model fitting and interpretation of the latent variable interaction difficult. This article compiles state-of-the-science techniques for assessing LMS model fit, obtaining standardized coefficients, and determining the size of the latent interaction effect in order to create a tutorial for new users of LMS models. The recommended sequence of model estimation and interpretation is demonstrated via a substantive example and a Monte Carlo simulation. Finally, extensions of this method are discussed, such as estimating quadratic effects of latent factors and interactions between latent slope and intercept factors, which hold significant potential for testing and advancing developmental theories.

Longitudinal Structural Equation Modeling: A Comprehensive Introduction

Abstract: 1. Review of Some Key Latent Variable Principles 2. Longitudinal Measurement Invariance 3. Structural Models for Comparing Dependent Means and Proportions 4. Fundamental Concepts of Stability and Change 5. Cross-Lagged Panel Models 6. Latent State-Trait Models 7. Linear Latent Growth Curve Models 8. Nonlinear Latent Growth Curve Models 9. Latent Difference Score Models 10. Latent Transition and Growth Mixture Models 11. Time Series Analysis 12. Survival Analysis Models 13. Missing Data and Attrition Appendix A: Notation Appendix B: A Primer on the Calculus of Change

Model‐based approaches to unconstrained ordination

Abstract: Summary Unconstrained ordination is commonly used in ecology to visualize multivariate data, in particular, to visualize the main trends between different sites in terms of their species composition or relative abundance. Methods of unconstrained ordination currently used, such as non-metric multidimensional scaling, are algorithm-based techniques developed and implemented without directly accommodating the statistical properties of the data at hand. Failure to account for these key data properties can lead to misleading results. A model-based approach to unconstrained ordination can address this issue, and in this study, two types of models for ordination are proposed based on finite mixture models and latent variable models. Each method is capable of handling different data types and different forms of species response to latent gradients. Further strengths of the models are demonstrated via example and simulation. Advantages of model-based approaches to ordination include the following: residual analysis tools for checking assumptions to ensure the fitted model is appropriate for the data; model selection tools to choose the most appropriate model for ordination; methods for formal statistical inference to draw conclusions from the ordination; and improved efficiency, that is model-based ordination better recovers true relationships between sites, when used appropriately.

Probabilistic slow feature analysis‐based representation learning from massive process data for soft sensor modeling

Abstract: Latent variable (LV) models provide explicit representations of underlying driving forces of process variations and retain the dominant information of process data In this study, slow features as temporally correlated LVs are derived using probabilistic slow feature analysis Slow features evolving in a state-space form effectively represent nominal variations of processes, some of which are potentially correlated to quality variables and hence help improving the prediction performance of soft sensors An efficient EM algorithm is proposed to estimate parameters of the probabilistic model, which turns out to be suitable for analyzing massive process data Two criteria are ∗To whom correspondence should be addressed †Tsinghua University ‡University of Alberta 1 also proposed to select quality-relevant slow features The validity and advantages of the proposed method are demonstrated via two case studies

Multi-conditional Latent Variable Model for Joint Facial Action Unit Detection

Abstract: We propose a novel multi-conditional latent variable model for simultaneous facial feature fusion and detection of facial action units. In our approach we exploit the structure-discovery capabilities of generative models such as Gaussian processes, and the discriminative power of classifiers such as logistic function. This leads to superior performance compared to existing classifiers for the target task that exploit either the discriminative or generative property, but not both. The model learning is performed via an efficient, newly proposed Bayesian learning strategy based on Monte Carlo sampling. Consequently, the learned model is robust to data overfitting, regardless of the number of both input features and jointly estimated facial action units. Extensive qualitative and quantitative experimental evaluations are performed on three publicly available datasets (CK+, Shoulder-pain and DISFA). We show that the proposed model outperforms the state-of-the-art methods for the target task on (i) feature fusion, and (ii) multiple facial action unit detection.

Group Factor Analysis

Abstract: Factor analysis (FA) provides linear factors that describe the relationships between individual variables of a data set. We extend this classical formulation into linear factors that describe the relationships between groups of variables, where each group represents either a set of related variables or a data set. The model also naturally extends canonical correlation analysis to more than two sets, in a way that is more flexible than previous extensions. Our solution is formulated as a variational inference of a latent variable model with structural sparsity, and it consists of two hierarchical levels: 1) the higher level models the relationships between the groups and 2) the lower models the observed variables given the higher level. We show that the resulting solution solves the group factor analysis (GFA) problem accurately, outperforming alternative FA-based solutions as well as more straightforward implementations of GFA. The method is demonstrated on two life science data sets, one on brain activation and the other on systems biology, illustrating its applicability to the analysis of different types of high-dimensional data sources.

Investigating Subjective and Objective Factors Influencing Teenagers' School Travel Mode Choice: Integrated Choice and Latent Variable Model

Abstract: In this paper, the authors apply Bhat and Dubey’s (2014) new multinomial probit (MNP)-based Integrated Choice Latent Variable (ICLV) formulation to analyze children’s travel mode choice to school. The new approach offered significant advantages, as it allowed the authors to incorporate three latent variables with a large data sample and with 10 ordinal indicators of the latent variables, and still estimate the ICLV model without any convergence problems. The data used in the empirical analysis originate from a survey undertaken in Cyprus in 2012. The results underscore the importance of incorporating subjective attitudinal variables in school mode choice modeling. The results also emphasize the need to improve bus and walking safety, and communicate such improvements to the public, especially to girls and women and high income households. The model application also provides important information regarding the value of investing in bicycling and walking infrastructure.

Investigating the subjective and objective factors influencing teenagers’ school travel mode choice – An integrated choice and latent variable model

Abstract: In this paper, we apply Bhat and Dubey’s (2014) new probit-kernel based Integrated Choice and Latent Variable (ICLV) model formulation to analyze children’s travel mode choice to school. The new approach offered significant advantages, as it allowed us to incorporate three latent variables with a large data sample and with 10 ordinal indicators of the latent variables, and still estimate the model without any convergence problems. The data used in the empirical analysis originates from a survey undertaken in Cyprus in 2012. The results underscore the importance of incorporating subjective attitudinal variables in school mode choice modeling. The results also emphasize the need to improve bus and walking safety, and communicate such improvements to the public, especially to girls and women and high income households. The model application also provides important information regarding the value of investing in bicycling and walking infrastructure.

Latent Variable Modeling with R

Abstract: This book demonstrates how to conduct latent variable modeling (LVM) in R by highlighting the features of each model, their specialized uses, examples, sample code and output, and an interpretation of the results Each chapter features a detailed example including the analysis of the data using R, the relevant theory, the assumptions underlying the model, and other statistical details to help readers better understand the models and interpret the results Every R command necessary for conducting the analyses is described along with the resulting output which provides readers with a template to follow when they apply the methods to their own data The basic information pertinent to each model, the newest developments in these areas, and the relevant R code to use them are reviewed Each chapter also features an introduction, summary, and suggested readings A glossary of the text’s boldfaced key terms and key R commands serve as helpful resources The book is accompanied by a website with exercises, an answer key, and the in-text example data sets Latent Variable Modeling with R: -Provides some examples that use messy data providing a more realistic situation readers will encounter with their own data -Reviews a wide range of LVMs including factor analysis, structural equation modeling, item response theory, and mixture models and advanced topics such as fitting nonlinear structural equation models, nonparametric item response theory models, and mixture regression models -Demonstrates how data simulation can help researchers better understand statistical methods and assist in selecting the necessary sample size prior to collecting data -wwwroutledgecom/9780415832458 provides exercises that apply the models along with annotated R output answer keys and the data that corresponds to the in-text examples so readers can replicate the results and check their work The book opens with basic instructions in how to use R to read data, download functions, and conduct basic analyses From there, each chapter is dedicated to a different latent variable model including exploratory and confirmatory factor analysis (CFA), structural equation modeling (SEM), multiple groups CFA/SEM, least squares estimation, growth curve models, mixture models, item response theory (both dichotomous and polytomous items), differential item functioning (DIF), and correspondance analysis The book concludes with a discussion of how data simulation can be used to better understand the workings of a statistical method and assist researchers in deciding on the necessary sample size prior to collecting data A mixture of independently developed R code along with available libraries for simulating latent models in R are provided so readers can use these simulations to analyze data using the methods introduced in the previous chapters Intended for use in graduate or advanced undergraduate courses in latent variable modeling, factor analysis, structural equation modeling, item response theory, measurement, or multivariate statistics taught in psychology, education, human development, and social and health sciences, researchers in these fields also appreciate this book’s practical approach The book provides sufficient conceptual background information to serve as a standalone text Familiarity with basic statistical concepts is assumed but basic knowledge of R is not

A framework for individualizing predictions of disease trajectories by exploiting multi-resolution structure

Abstract: For many complex diseases, there is a wide variety of ways in which an individual can manifest the disease. The challenge of personalized medicine is to develop tools that can accurately predict the trajectory of an individual's disease, which can in turn enable clinicians to optimize treatments. We represent an individual's disease trajectory as a continuous-valued continuous-time function describing the severity of the disease over time. We propose a hierarchical latent variable model that individualizes predictions of disease trajectories. This model shares statistical strength across observations at different resolutions-the population, subpopulation and the individual level. We describe an algorithm for learning population and subpopulation parameters offline, and an online procedure for dynamically learning individual-specific parameters. Finally, we validate our model on the task of predicting the course of interstitial lung disease, a leading cause of death among patients with the autoimmune disease scleroderma. We compare our approach against state-of-the-art and demonstrate significant improvements in predictive accuracy.

Residual Associations in Latent Class and Latent Transition Analysis

Abstract: This article explores a method for modeling associations among binary and ordered categorical variables. The method has the advantage that maximum-likelihood estimation can be used in multivariate models without numerical integration because the observed data log-likelihood has an explicit form. The association model is especially useful with mixture models to handle violations of the local independence assumption. Applications to latent class and latent transition analysis are presented.

A Comparison of Imputation Strategies for Ordinal Missing Data on Likert Scale Variables

Abstract: This article compares a variety of imputation strategies for ordinal missing data on Likert scale variables (number of categories = 2, 3, 5, or 7) in recovering reliability coefficients, mean scale scores, and regression coefficients of predicting one scale score from another. The examined strategies include imputing using normal data models with naive rounding/without rounding, using latent variable models, and using categorical data models such as discriminant analysis and binary logistic regression (for dichotomous data only), multinomial and proportional odds logistic regression (for polytomous data only). The result suggests that both the normal model approach without rounding and the latent variable model approach perform well for either dichotomous or polytomous data regardless of sample size, missing data proportion, and asymmetry of item distributions. The discriminant analysis approach also performs well for dichotomous data. Naively rounding normal imputations or using logistic regression models to impute ordinal data are not recommended as they can potentially lead to substantial bias in all or some of the parameters.

HMM-Driven Robust Probabilistic Principal Component Analyzer for Dynamic Process Fault Classification

Abstract: In this paper, a novel hidden Markov model (HMM)-driven robust latent variable model (LVM) is proposed for fault classification in dynamic industrial processes. A robust probabilistic model with Student's $t$ mixture output is designed for tolerating outliers. Based on the robust LVM, the probabilistic structure is further developed into a classifier form so as to incorporate various types of process information during model acquisition. After that, the robust probabilistic classifier is extended within the HMM framework so as to characterize the time-domain stochastic uncertainties. The model parameters are derived through the expectation–maximization algorithm. For performance validation, the developed model is tested on the Tennessee Eastman benchmark process.

Dynamic Estimation of Latent Opinion Using a Hierarchical Group-Level IRT Model

Abstract: Over the past eight decades, millions of people have been surveyed on their political opinions. Until recently, however, polls rarely included enough questions in a given domain to apply scaling techniques such as IRT models at the individual level, preventing scholars from taking full advantage of historical survey data. To address this problem, we develop a Bayesian group-level IRT approach that models latent traits at the level of demographic and/or geographic groups rather than individuals. We use a hierarchical model to borrow strength cross-sectionally and dynamic linear models to do so across time. The group-level estimates can be weighted to generate estimates for geographic units. This framework opens up vast new areas of research on historical public opinion, especially at the subnational level. We illustrate this potential by estimating the average policy liberalism of citizens in each U.S. state in each year between 1972 and 2012.

Factor copula models for item response data.

Abstract: Factor or conditional independence models based on copulas are proposed for multivariate discrete data such as item responses. The factor copula models have interpretations of latent maxima/minima (in comparison with latent means) and can lead to more probability in the joint upper or lower tail compared with factor models based on the discretized multivariate normal distribution (or multidimensional normal ogive model). Details on maximum likelihood estimation of parameters for the factor copula model are given, as well as analysis of the behavior of the log-likelihood. Our general methodology is illustrated with several item response data sets, and it is shown that there is a substantial improvement on existing models both conceptually and in fit to data.

RAND-WALK: A Latent Variable Model Approach to Word Embeddings

Abstract: Semantic word embeddings represent the meaning of a word via a vector, and are created by diverse methods. Many use nonlinear operations on co-occurrence statistics, and have hand-tuned hyperparameters and reweighting methods. This paper proposes a new generative model, a dynamic version of the log-linear topic model of~\citet{mnih2007three}. The methodological novelty is to use the prior to compute closed form expressions for word statistics. This provides a theoretical justification for nonlinear models like PMI, word2vec, and GloVe, as well as some hyperparameter choices. It also helps explain why low-dimensional semantic embeddings contain linear algebraic structure that allows solution of word analogies, as shown by~\citet{mikolov2013efficient} and many subsequent papers. Experimental support is provided for the generative model assumptions, the most important of which is that latent word vectors are fairly uniformly dispersed in space.

Multilayer Joint Gait-Pose Manifolds for Human Gait Motion Modeling

Abstract: We present new multilayer joint gait-pose manifolds (multilayer JGPMs) for complex human gait motion modeling, where three latent variables are defined jointly in a low-dimensional manifold to represent a variety of body configurations. Specifically, the pose variable (along the pose manifold) denotes a specific stage in a walking cycle; the gait variable (along the gait manifold) represents different walking styles; and the linear scale variable characterizes the maximum stride in a walking cycle. We discuss two kinds of topological priors for coupling the pose and gait manifolds, i.e., cylindrical and toroidal, to examine their effectiveness and suitability for motion modeling. We resort to a topologically-constrained Gaussian process (GP) latent variable model to learn the multilayer JGPMs where two new techniques are introduced to facilitate model learning under limited training data. First is training data diversification that creates a set of simulated motion data with different strides. Second is the topology-aware local learning to speed up model learning by taking advantage of the local topological structure. The experimental results on the Carnegie Mellon University motion capture data demonstrate the advantages of our proposed multilayer models over several existing GP-based motion models in terms of the overall performance of human gait motion modeling.

Rasch Mixture Models for DIF Detection: A Comparison of Old and New Score Specifications.

Abstract: Rasch mixture models can be a useful tool when checking the assumption of measurement invariance for a single Rasch model. They provide advantages compared to manifest differential item functioning (DIF) tests when the DIF groups are only weakly correlated with the manifest covariates available. Unlike in single Rasch models, estimation of Rasch mixture models is sensitive to the specification of the ability distribution even when the conditional maximum likelihood approach is used. It is demonstrated in a simulation study how differences in ability can influence the latent classes of a Rasch mixture model. If the aim is only DIF detection, it is not of interest to uncover such ability differences as one is only interested in a latent group structure regarding the item difficulties. To avoid any confounding effect of ability differences (or impact), a new score distribution for the Rasch mixture model is introduced here. It ensures the estimation of the Rasch mixture model to be independent of the ability distribution and thus restricts the mixture to be sensitive to latent structure in the item difficulties only. Its usefulness is demonstrated in a simulation study, and its application is illustrated in a study of verbal aggression.

Margins of discrete Bayesian networks

Abstract: Bayesian network models with latent variables are widely used in statistics and machine learning. In this paper we provide a complete algebraic characterization of Bayesian network models with latent variables when the observed variables are discrete and no assumption is made about the state-space of the latent variables. We show that it is algebraically equivalent to the so-called nested Markov model, meaning that the two are the same up to inequality constraints on the joint probabilities. In particular these two models have the same dimension. The nested Markov model is therefore the best possible description of the latent variable model that avoids consideration of inequalities, which are extremely complicated in general. A consequence of this is that the constraint finding algorithm of Tian and Pearl (UAI 2002, pp519-527) is complete for finding equality constraints. Latent variable models suffer from difficulties of unidentifiable parameters and non-regular asymptotics; in contrast the nested Markov model is fully identifiable, represents a curved exponential family of known dimension, and can easily be fitted using an explicit parameterization.

Relating Latent Change Score and Continuous Time Models

Abstract: The primary goal of this article is to demonstrate the close relationship between 2 classes of dynamic models in psychological research: latent change score models and continuous time models. The secondary goal is to point out some differences. We begin with a brief review of both approaches, before demonstrating how the 2 methods are mathematically and conceptually related. It will be shown that most commonly used latent change score models are related to continuous time models by the difference equation approximation to the differential equation. One way in which the 2 approaches differ is the treatment of time. Whereas there are theoretical and practical restrictions regarding observation time points and intervals in latent change score models, no such limitations exist in continuous time models. We illustrate our arguments with three simulated data sets using a univariate and bivariate model with equal and unequal time intervals. As a by-product of this comparison, we discuss the use of phantom and de...

Testing and Modeling Dependencies Between a Network and Nodal Attributes.

Abstract: Network analysis is often focused on characterizing the dependencies between network relations and node-level attributes. Potential relationships are typically explored by modeling the network as a function of the nodal attributes or by modeling the attributes as a function of the network. These methods require specification of the exact nature of the association between the network and attributes, reduce the network data to a small number of summary statistics, and are unable to provide predictions simultaneously for missing attribute and network information. Existing methods that model the attributes and network jointly also assume the data are fully observed. In this article, we introduce a unified approach to analysis that addresses these shortcomings. We use a previously developed latent variable model to obtain a low-dimensional representation of the network in terms of node-specific network factors. We introduce a novel testing procedure to determine if dependencies exist between the network factor...

A discrete time event-history approach to informative drop-out in mixed latent Markov models with covariates.

Abstract: Summary Mixed latent Markov (MLM) models represent an important tool of analysis of longitudinal data when response variables are affected by time-fixed and time-varying unobserved heterogeneity, in which the latter is accounted for by a hidden Markov chain. In order to avoid bias when using a model of this type in the presence of informative drop-out, we propose an event-history (EH) extension of the latent Markov approach that may be used with multivariate longitudinal data, in which one or more outcomes of a different nature are observed at each time occasion. The EH component of the resulting model is referred to the interval-censored drop-out, and bias in MLM modeling is avoided by correlated random effects, included in the different model components, which follow common latent distributions. In order to perform maximum likelihood estimation of the proposed model by the expectation–maximization algorithm, we extend the usual forward-backward recursions of Baum and Welch. The algorithm has the same complexity as the one adopted in cases of non-informative drop-out. We illustrate the proposed approach through simulations and an application based on data coming from a medical study about primary biliary cirrhosis in which there are two outcomes of interest, one continuous and the other binary.

Online tensor methods for learning latent variable models

Abstract: We introduce an online tensor decomposition based approach for two latent variable modeling problems namely, (1) community detection, in which we learn the latent communities that the social actors in social networks belong to, and (2) topic modeling, in which we infer hidden topics of text articles. We consider decomposition of moment tensors using stochastic gradient descent. We conduct optimization of multilinear operations in SGD and avoid directly forming the tensors, to save computational and storage costs. We present optimized algorithm in two platforms. Our GPU-based implementation exploits the parallelism of SIMD architectures to allow for maximum speed-up by a careful optimization of storage and data transfer, whereas our CPU-based implementation uses efficient sparse matrix computations and is suitable for large sparse data sets. For the community detection problem, we demonstrate accuracy and computational efficiency on Facebook, Yelp and DBLP data sets, and for the topic modeling problem, we also demonstrate good performance on the New York Times data set. We compare our results to the state-of-the-art algorithms such as the variational method, and report a gain of accuracy and a gain of several orders of magnitude in the execution time.

Introducing non-normality of latent psychological constructs in choice modeling with an application to bicyclist route choice

Abstract: In the current paper, we propose the use of a multivariate skew-normal (MSN) distribution function for the latent psychological constructs within the context of an integrated choice and latent variable (ICLV) model system. The multivariate skew-normal (MSN) distribution that we use is tractable, parsimonious in parameters that regulate the distribution and its skewness, and includes the normal distribution as a special interior point case (this allows for testing with the traditional ICLV model). Our procedure to accommodate non-normality in the psychological constructs exploits the latent factor structure of the ICLV model, and is a flexible, yet very efficient approach (through dimension-reduction) to accommodate a multivariate non-normal structure across all indicator and outcome variables in a multivariate system through the specification of a much lower-dimensional multivariate skew-normal distribution for the structural errors. Taste variations (i.e., heterogeneity in sensitivity to response variables) can also be introduced efficiently and in a non-normal fashion through interactions of explanatory variables with the latent variables. The resulting model we develop is suitable for estimation using Bhat’s (2011) maximum approximate composite marginal likelihood (MACML) inference approach. The proposed model is applied to model bicyclists’ route choice behavior using a web-based survey of Texas bicyclists. The results reveal evidence for non-normality in the latent constructs. From a substantive point of view, the results suggest that the most unattractive features of a bicycle route are long travel times (for commuters), heavy motorized traffic volume, absence of a continuous bicycle facility, and high parking occupancy rates and long lengths of parking zones along the route.

MANTRA: Minimum Maximum Latent Structural SVM for Image Classification and Ranking

Abstract: In this work, we propose a novel Weakly Supervised Learning (WSL) framework dedicated to learn discriminative part detectors from images annotated with a global label. Our WSL method encompasses three main contributions. Firstly, we introduce a new structured output latent variable model, Minimum mAximum lateNt sTRucturAl SVM (MANTRA), which prediction relies on a pair of latent variables: h+ (resp. h-) provides positive (resp. negative) evidence for a given output y. Secondly, we instantiate MANTRA for two different visual recognition tasks: multi-class classification and ranking. For ranking, we propose efficient solutions to exactly solve the inference and the loss-augmented problems. Finally, extensive experiments highlight the relevance of the proposed method: MANTRA outperforms state-of-the art results on five different datasets.

A Direct Latent Variable Modeling Based Method for Point and Interval Estimation of Coefficient Alpha.

Abstract: A direct approach to point and interval estimation of Cronbach’s coefficient alpha for multiple component measuring instruments is outlined. The procedure is based on a latent variable modeling application with widely circulated software. As a by-product, using sample data the method permits ascertaining whether the population discrepancy between alpha and the composite reliability coefficient may be practically negligible for a given empirical setting. The outlined approach is illustrated with numerical data.