Showing papers in "Multivariate Behavioral Research in 2011"

An Introduction to Propensity Score Methods for Reducing the Effects of Confounding in Observational Studies

Abstract: The propensity score is the probability of treatment assignment conditional on observed baseline characteristics. The propensity score allows one to design and analyze an observational (nonrandomized) study so that it mimics some of the particular characteristics of a randomized controlled trial. In particular, the propensity score is a balancing score: conditional on the propensity score, the distribution of observed baseline covariates will be similar between treated and untreated subjects. I describe 4 different propensity score methods: matching on the propensity score, stratification on the propensity score, inverse probability of treatment weighting using the propensity score, and covariate adjustment using the propensity score. I describe balance diagnostics for examining whether the propensity score model has been adequately specified. Furthermore, I discuss differences between regression-based methods and propensity score-based methods for the analysis of observational data. I describe different causal average treatment effects and their relationship with propensity score analyses.

Bias in Cross-Sectional Analyses of Longitudinal Mediation: Partial and Complete Mediation Under an Autoregressive Model

Abstract: Maxwell and Cole (2007) showed that cross-sectional approaches to mediation typically generate substantially biased estimates of longitudinal parameters in the special case of complete mediation. However, their results did not apply to the more typical case of partial mediation. We extend their previous work by showing that substantial bias can also occur with partial mediation. In particular, cross-sectional analyses can imply the existence of a substantial indirect effect even when the true longitudinal indirect effect is zero. Thus, a variable that is found to be a strong mediator in a cross-sectional analysis may not be a mediator at all in a longitudinal analysis. In addition, we show that very different combinations of longitudinal parameter values can lead to essentially identical cross-sectional correlations, raising serious questions about the interpretability of cross-sectional mediation data. More generally, researchers are encouraged to consider a wide variety of possible mediation models beyond simple cross-sectional models, including but not restricted to autoregressive models of change.

A Systematic Review of Propensity Score Methods in the Social Sciences.

Abstract: The use of propensity scores in psychological and educational research has been steadily increasing in the last 2 to 3 years. However, there are some common misconceptions about the use of different estimation techniques and conditioning choices in the context of propensity score analysis. In addition, reporting practices for propensity score analyses often lack important details that allow other researchers to confidently judge the appropriateness of reported analyses and potentially to replicate published findings. In this article we conduct a systematic literature review of a large number of published articles in major areas of social science that used propensity scores up until the fall of 2009. We identify common errors in estimation, conditioning, and reporting of propensity score analyses and suggest possible solutions.

A Tutorial and Case Study in Propensity Score Analysis: An Application to Estimating the Effect of In-Hospital Smoking Cessation Counseling on Mortality

Abstract: Propensity score methods allow investigators to estimate causal treatment effects using observational or nonrandomized data. In this article we provide a practical illustration of the appropriate steps in conducting propensity score analyses. For illustrative purposes, we use a sample of current smokers who were discharged alive after being hospitalized with a diagnosis of acute myocardial infarction. The exposure of interest was receipt of smoking cessation counseling prior to hospital discharge and the outcome was mortality with 3 years of hospital discharge. We illustrate the following concepts: first, how to specify the propensity score model; second, how to match treated and untreated participants on the propensity score; third, how to compare the similarity of baseline characteristics between treated and untreated participants after stratifying on the propensity score, in a sample matched on the propensity score, or in a sample weighted by the inverse probability of treatment; fourth, how to estimat...

The Hull Method for Selecting the Number of Common Factors

Abstract: A common problem in exploratory factor analysis is how many factors need to be extracted from a particular data set. We propose a new method for selecting the number of major common factors: the Hull method, which aims to find a model with an optimal balance between model fit and number of parameters. We examine the performance of the method in an extensive simulation study in which the simulated data are based on major and minor factors. The study compares the method with four other methods such as parallel analysis and the minimum average partial test, which were selected because they have been proven to perform well and/or they are frequently used in applied research. The Hull method outperformed all four methods at recovering the correct number of major factors. Its usefulness was further illustrated by its assessment of the dimensionality of the Five-Factor Personality Inventory ( Hendriks, Hofstee, & De Raad, 1999 ). This inventory has 100 items, and the typical methods for assessing dimensionality prove to be useless: the large number of factors they suggest has no theoretical justification. The Hull method, however, suggested retaining the number of factors that the theoretical background to the inventory actually proposes.

Commentary: Mediation Analysis, Causal Process, and Cross-Sectional Data.

Abstract: Maxwell, Cole, and Mitchell (2011) extended the work of Maxwell and Cole (2007) , which raised important questions about whether mediation analyses based on cross-sectional data can shed light on longitudinal mediation process. The latest article considers longitudinal processes that can only be partially explained by an intervening variable, and Maxwell et al. showed that the same general conclusions are obtained, namely that analyses of cross-sectional data will not reveal the longitudinal mediation process. While applauding the advances of the target article, this comment encourages the detailed exploration of alternate causal models in psychology beyond the autoregressive model considered by Maxwell et al. When inferences based on cross-sectional analyses are compared to alternate models, different patterns of bias are likely to be observed. I illustrate how different models of the causal process can be derived using examples from research on psychopathology.

Accuracy in Parameter Estimation for the Root Mean Square Error of Approximation: Sample Size Planning for Narrow Confidence Intervals.

Abstract: The root mean square error of approximation (RMSEA) is one of the most widely reported measures of misfit/fit in applications of structural equation modeling. When the RMSEA is of interest, so too should be the accompanying confidence interval. A narrow confidence interval reveals that the plausible parameter values are confined to a relatively small range at the specified level of confidence. The accuracy in parameter estimation approach to sample size planning is developed for the RMSEA so that the confidence interval for the population RMSEA will have a width whose expectation is sufficiently narrow. Analytic developments are shown to work well with a Monte Carlo simulation study. Freely available computer software is developed so that the methods discussed can be implemented. The methods are demonstrated for a repeated measures design where the way in which social relationships and initial depression influence coping strategies and later depression are examined.

The Longitudinal Interplay of Adolescents' Self-Esteem and Body Image: A Conditional Autoregressive Latent Trajectory Analysis

Abstract: Self-esteem and body image are central to coping successfully with the developmental challenges of adolescence. However, the current knowledge surrounding self-esteem and body image is fraught with controversy. This study attempts to clarify some of them by addressing three questions: (1) Are the intraindividual developmental trajectories of self-esteem and body image stable across adolescence? (2) What is the direction of the relations between body image and self-esteem over time? (3) What is the role of gender, ethnicity, and pubertal development on those trajectories? This study relies on Autoregressive Latent Trajectory analyses based on data from a 4-year, 6-wave, prospective longitudinal study of 1,001 adolescents. Self-esteem and body image levels remained high and stable over time, although body image levels also tended to increase slightly. The results show that levels of self-esteem were positively influenced by levels of body image. However, these effects remained small and most of the observed...

The Use of Propensity Scores for Nonrandomized Designs With Clustered Data

Abstract: In this article we propose several modeling choices to extend propensity score analysis to clustered data. We describe different possible model specifications for estimation of the propensity score...

Multilevel SEM Strategies for Evaluating Mediation in Three-Level Data

Abstract: Strategies for modeling mediation effects in multilevel data have proliferated over the past decade, keeping pace with the demands of applied research. Approaches for testing mediation hypotheses with 2-level clustered data were first proposed using multilevel modeling (MLM) and subsequently using multilevel structural equation modeling (MSEM) to overcome several limitations of MLM. Because 3-level clustered data are becoming increasingly common, it is necessary to develop methods to assess mediation in such data. Whereas MLM easily accommodates 3-level data, MSEM does not. However, it is possible to specify and estimate some 3-level mediation models using both single- and multilevel SEM. Three new alternative approaches are proposed for fitting 3-level mediation models using single- and multilevel SEM, and each method is demonstrated with simulated data. Discussion focuses on the advantages and disadvantages of these approaches as well as directions for future research.

On Inclusion of Covariates for Class Enumeration of Growth Mixture Models

Abstract: In this article, we directly questioned the common practice in growth mixture model (GMM) applications that exclusively rely on the fitting model without covariates for GMM class enumeration. We provided theoretical and simulation evidence to demonstrate that exclusion of covariates from GMM class enumeration could be problematic in many cases. Based on our findings, we provided recommendations for examining the class enumeration by the fitting model without covariates and discussed the potential of covariate inclusion as a remedy for the weakness of GMM class enumeration without including covariates. A real example on the development of children's cumulative exposure to risk factors for adolescent substance use was provided to illustrate our methodological developments.

The use of propensity scores in mediation analysis

Abstract: Mediation analysis uses measures of hypothesized mediating variables to test theory for how a treatment achieves effects on outcomes and to improve subsequent treatments by identifying the most eff...

Dynamic Factor Analysis Models With Time-Varying Parameters

Abstract: Dynamic factor analysis models with time-varying parameters offer a valuable tool for evaluating multivariate time series data with time-varying dynamics and/or measurement properties. We use the Dynamic Model of Activation proposed by Zautra and colleagues (Zautra, Potter, & Reich, 1997) as a motivating example to construct a dynamic factor model with vector autoregressive relations and time-varying cross-regression parameters at the factor level. Using techniques drawn from the state-space literature, the model was fitted to a set of daily affect data (over 71 days) from 10 participants who had been diagnosed with Parkinson's disease. Our empirical results lend partial support and some potential refinement to the Dynamic Model of Activation with regard to how the time dependencies between positive and negative affects change over time. A simulation study is conducted to examine the performance of the proposed techniques when (a) changes in the time-varying parameters are represented using the true model of change, (b) supposedly time-invariant parameters are represented as time-varying, and

Latent Differential Equation Modeling of Self-Regulatory and Coregulatory Affective Processes

Abstract: We examine emotion self-regulation and coregulation in romantic couples using daily self-reports of positive and negative affect. We fit these data using a damped linear oscillator model specified as a latent differential equation to investigate affect dynamics at the individual level and coupled influences for the 2 partners in each couple. Results indicate an absence of damping of relationship-specific affect within individuals in the sample. When both positive and negative affect are modeled at the individual level, the influence of positive affect is greater than that of negative affect. At the dyad level, the findings indicate coupled influences in both positive and negative affect between partners. With regard to positive affect, females are sensitive to their partners' overall displacement from average as well as their rate of change; males are sensitive only to their partners' displacement from average. For negative affect both partners are sensitive to each other's displacement from average, yet there are no coupled influences for rates of change in this dimension. We interpret the influence of the parameters on the system by examining the expected behavior of the system as a function of varying parameter values.

Challenges With Propensity Score Strategies in a High-Dimensional Setting and a Potential Alternative

Abstract: This article explores some of the challenges that arise when trying to implement propensity score strategies to answer a causal question using data with a large number of covariates. We discuss choices in propensity score estimation strategies, matching and weighting implementation strategies, balance diagnostics, and final analysis models. We demonstrate the wide range of estimates that can result from different combinations of these choices. Finally, an alternative estimation strategy is presented that may have benefits in terms of simplicity and reliability. These issues are explored in the context of an empirical example that uses data from the Early Childhood Longitudinal Study, Kindergarten Cohort to investigate the potential effect of grade retention after the 1st-grade year on subsequent cognitive outcomes.

Sensitivity Analysis in Structural Equation Models: Cases and Their Influence.

Abstract: The detection of outliers and influential observations is routine practice in linear regression. Despite ongoing extensions and development of case diagnostics in structural equation models (SEM), their application has received limited attention and understanding in practice. The use of case diagnostics informs analysts of the uncertainty of model estimates under different subsets of the data and highlights unusual and important characteristics of certain cases. We present several measures of case influence applicable in SEM and illustrate their implementation, presentation, and interpretation with two empirical examples: (a) a common factor model on verbal and visual ability (Holzinger & Swineford, 1939) and (b) a general structural equation model assessing the effect of industrialization on democracy in a mediating model using country-level data (Bollen, 1989; Bollen & Arminger, 1991). Throughout these examples, three issues are emphasized. First, cases may impact different aspects of results as identif...

Bias and Efficiency in Structural Equation Modeling: Maximum Likelihood Versus Robust Methods.

Abstract: In the structural equation modeling literature, the normal-distribution-based maximum likelihood (ML) method is most widely used, partly because the resulting estimator is claimed to be asymptotically unbiased and most efficient. However, this may not hold when data deviate from normal distribution. Outlying cases or nonnormally distributed data, in practice, can make the ML estimator (MLE) biased and inefficient. In addition to ML, robust methods have also been developed, which are designed to minimize the effects of outlying cases. But the properties of robust estimates and their standard errors (SEs) have never been systematically studied. This article studies two robust methods and compares them against the ML method with respect to bias and efficiency using a confirmatory factor model. Simulation results show that robust methods lead to results comparable with ML when data are normally distributed. When data have heavy tails or outlying cases, robust methods lead to less biased and more efficient est...

Commentary: Using Potential Outcomes to Understand Causal Mediation Analysis

Abstract: In this commentary, we demonstrate how the potential outcomes framework can help understand the key identification assumptions underlying causal mediation analysis. We show that this framework can lead to the development of alternative research design and statistical analysis strategies applicable to the longitudinal data settings considered by Maxwell, Cole, and Mitchell (2011).

Bayesian Inference for Growth Mixture Models with Latent Class Dependent Missing Data.

Abstract: Growth mixture models (GMMs) with nonignorable missing data have drawn increasing attention in research communities but have not been fully studied. The goal of this article is to propose and to evaluate a Bayesian method to estimate the GMMs with latent class dependent missing data. An extended GMM is first presented in which class probabilities depend on some observed explanatory variables and data missingness depends on both the explanatory variables and a latent class variable. A full Bayesian method is then proposed to estimate the model. Through the data augmentation method, conditional posterior distributions for all model parameters and missing data are obtained. A Gibbs sampling procedure is then used to generate Markov chains of model parameters for statistical inference. The application of the model and the method is first demonstrated through the analysis of mathematical ability growth data from the National Longitudinal Survey of Youth 1997 (Bureau of Labor Statistics, U.S. Department of Labor, 1997). A simulation study considering 3 main factors (the sample size, the class probability, and the missing data mechanism) is then conducted and the results show that the proposed Bayesian estimation approach performs very well under the studied conditions. Finally, some implications of this study, including the misspecified missingness mechanism, the sample size, the sensitivity of the model, the number of latent classes, the model comparison, and the future directions of the approach, are discussed.

Commentary: Are Three Waves of Data Sufficient for Assessing Mediation?

Abstract: Maxwell, Cole, and Mitchell (2011) demonstrated that simple structural equation models, when used with cross-sectional data, generally produce biased estimates of meditated effects. I extend those results by showing how simple structural equation models can produce biased estimates of meditated effects when used even with longitudinal data. Even with longitudinal data, simple autoregressive structural equation models can imply the existence of indirect effects when only direct effects exist and the existence of direct effects when only indirect effects exist.

Comparison of Modern Methods for Analyzing Repeated Measures Data With Missing Values

Abstract: Missing data are a pervasive problem in many psychological applications in the real world. In this article we study the impact of dropout on the operational characteristics of several approaches that can be easily implemented with commercially available software. These approaches include the covariance pattern model based on an unstructured covariance matrix (CPM-U) and the true covariance matrix (CPM-T), multiple imputation-based generalized estimating equations (MI-GEE), and weighted generalized estimating equations (WGEE). Under the missing at random mechanism, the MI-GEE approach was always robust. The CPM-T and CPM-U methods were also able to control the error rates provided that certain minimum sample size requirements were met, whereas the WGEE was more prone to inflated error rates. In contrast, under the missing not at random mechanism, all evaluated approaches were generally invalid. Our results also indicate that the CPM methods were more powerful than the MI-GEE and WGEE methods and their superiority was often substantial. Furthermore, we note that little or no power was sacrificed by using CPM-U method in place of CPM-T, although both methods have less power in situations where some participants have incomplete data. Some aspects of the CPM-U and MI-GEE methods are illustrated using real data from 2 previously published data sets. The first data set comes from a randomized study of AIDS patients with advanced immune suppression, the second from a cohort of patients with schizotypal personality disorder enrolled in a prevention program for psychosis.

Conceptualizing and Estimating Process Speed in Studies Employing Ecological Momentary Assessment Designs: A Multilevel Variance Decomposition Approach

Abstract: Researchers have been making use of ecological momentary assessment (EMA) and other study designs that sample feelings and behaviors in real time and in naturalistic settings to study temporal dynamics and contextual factors of a wide variety of psychological, physiological, and behavioral processes. As EMA designs become more widespread, questions are arising about the frequency of data sampling, with direct implications for participants' burden and researchers' ability to capture and study dynamic processes. Traditionally, spectral analytic techniques are used for time series data to identify process speed. However, the nature of EMA data, often collected with fewer than 100 measurements per person, sampled at randomly spaced intervals, and replete with planned and unplanned missingness, precludes application of traditional spectral analytic techniques. Building on principles of variance partitioning used in the generalizability theory of measurement and spectral analysis, we illustrate the utility of m...

Scaling Variances of Latent Variables by Standardizing Loadings: Applications to Working Memory and the Position Effect.

Abstract: The standardization of loadings gives a metric to the corresponding latent variable and thus scales the variance of this latent variable. By assigning an appropriately estimated weight to all the loadings on the same latent variable it can be achieved that the average squared loading is 1 as the result of standardization. As a consequence, there is comparability of the variances of the latent variables of a confirmatory factor model. A precondition of comparability is that the latent variables must have loadings of the same manifest variables and that the variances are estimated with respect to the same covariance matrix. The usefulness of this standardization method is demonstrated by applying it for the evaluation of the sources of performance in a working memory task and for the evaluation of the impact of the position effect on performance in completing a reasoning measure. In these examples the scaled variances of the latent variables provided useful information.

Model Selection with the Linear Mixed Model for Longitudinal Data.

Abstract: Model building or model selection with linear mixed models (LMMs) is complicated by the presence of both fixed effects and random effects. The fixed effects structure and random effects structure are codependent, so selection of one influences the other. Most presentations of LMM in psychology and education are based on a multilevel or hierarchical approach in which the variance-covariance matrix of the random effects is assumed to be positive definite with nonzero values for the variances. When the number of fixed effects and random effects is unknown, the predominant approach to model building is a step-up method in which one starts with a limited model (e.g., few fixed and random intercepts) and then additional fixed effects and random effects are added based on statistical tests. A model building approach that has received less attention in psychology and education is a top-down method. In the top-down method, the initial model has a single random intercept but is loaded with fixed effects (also known as an "overelaborate" model). Based on the overelaborate fixed effects model, the need for additional random effects is determined. There has been little if any examination of the ability of these methods to identify a true population model (i.e., identifying the model that generated the data). The purpose of this article is to examine the performance of the step-up and top-down model building approaches for exploratory longitudinal data analysis. Student achievement data sets from the Chicago longitudinal study serve as the populations in the simulations.

Comparison of Methods for Collecting and Modeling Dissimilarity Data: Applications to Complex Sound Stimuli

Abstract: Sorting procedures are frequently adopted as an alternative to dissimilarity ratings to measure the dissimilarity of large sets of stimuli in a comparatively short time. However, systematic empirical research on the consequences of this experiment-design choice is lacking. We carried out a behavioral experiment to assess the extent to which sorting procedures compare to dissimilarity ratings in terms of efficiency, reliability, and accuracy, and the extent to which data from different data-collection methods are redundant and are better fit by different distance models. Participants estimated the dissimilarity of either semantically charged environmental sounds or semantically neutral synthetic sounds. We considered free and hierarchical sorting and derived indications concerning the properties of constrained and truncated hierarchical sorting methods from hierarchical sorting data. Results show that the higher efficiency of sorting methods comes at a considerable cost in terms of data reliability and accuracy. This loss appears to be minimized with truncated hierarchical sorting methods that start from a relatively low number of groups of stimuli. Finally, variations in data-collection method differentially affect the fit of various distance models at the group-average and individual levels. On the basis of these results, we suggest adopting sorting as an alternative to dissimilarity-rating methods only when strictly necessary. We also suggest analyzing the raw behavioral dissimilarities, and avoiding modeling them with one single distance model.

Covariate Selection in Propensity Scores Using Outcome Proxies.

Abstract: This study examined the practical problem of covariate selection in propensity scores (PSs) given a predetermined set of covariates. Because the bias reduction capacity of a confounding covariate i...

Identification of Differential Item Functioning in Multiple-Group Settings: A Multivariate Outlier Detection Approach.

Abstract: We focus on the identification of differential item functioning (DIF) when more than two groups of examinees are considered. We propose to consider items as elements of a multivariate space, where DIF items are outlying elements. Following this approach, the situation of multiple groups is a quite natural case. A robust statistics technique is proposed to identify DIF items as outliers in the multivariate space. For low dimensionalities, up to 2-3 groups, a simple graphical tool is derived. We illustrate our approach with a reanalysis of data from Kim, Cohen, and Park (1995) on using calculators for a mathematics test.

Investigating the Nature of Method Factors Through Multiple Informants: Evidence for a Specific Factor?

Abstract: When a self-report instrument includes a balanced number of positively and negatively worded items, factor analysts often use method factors to aid model fitting The nature of these factors, often referred to as acquiescence, is still debated Relying upon previous results (Alessandri et al, 2010; DiStefano & Motl, 2006, 2008; Rauch, Schweizer, & Moosbrugger, 2007), we submit that the so-called method factors, instead, represent substantive specific factors This study investigates the convergence of method effects across different observers The revised Life Orientation Test (Scheier, Carver, & Bridges, 1994) was administered to a sample of 372 adults (57% females), with 372 acquaintances serving as informants Results showed that a specific factor was detectable both with self- and other-ratings A significant correlation across informants provided evidence for the convergence of this specific factor Construct validity was examined by locating this specific factor within a nomological net of personal

A Comparison of Paper and Online Tests Using a within-Subjects Design and Propensity Score Matching Study

Abstract: This inquiry had 2 components: (1) the first was substantive and focused on the comparability of paper-based and computer-based test forms and (2) the second was a within-study comparison wherein a...