Showing papers in "Psychological Methods in 2014"

Reliability Estimation in a Multilevel Confirmatory Factor Analysis Framework

Abstract: Scales with varying degrees of measurement reliability are often used in the context of multistage sampling, where variance exists at multiple levels of analysis (e.g., individual and group). Because methodological guidance on assessing and reporting reliability at multiple levels of analysis is currently lacking, we discuss the importance of examining level-specific reliability. We present a simulation study and an applied example showing different methods for estimating multilevel reliability using multilevel confirmatory factor analysis and provide supporting Mplus program code. We conclude that (a) single-level estimates will not reflect a scale's actual reliability unless reliability is identical at each level of analysis, (b) 2-level alpha and composite reliability (omega) perform relatively well in most settings, (c) estimates of maximal reliability (H) were more biased when estimated using multilevel data than either alpha or omega, and (d) small cluster size can lead to overestimates of reliability at the between level of analysis. We also show that Monte Carlo confidence intervals and Bayesian credible intervals closely reflect the sampling distribution of reliability estimates under most conditions. We discuss the estimation of credible intervals using Mplus and provide R code for computing Monte Carlo confidence intervals.

Modeling dependent effect sizes with three-level meta-analyses: a structural equation modeling approach.

Abstract: Meta-analysis is an indispensable tool used to synthesize research findings in the social, educational, medical, management, and behavioral sciences. Most meta-analytic models assume independence among effect sizes. However, effect sizes can be dependent for various reasons. For example, studies might report multiple effect sizes on the same construct, and effect sizes reported by participants from the same cultural group are likely to be more similar than those reported by other cultural groups. This article reviews the problems and common methods to handle dependent effect sizes. The objective of this article is to demonstrate how 3-level meta-analyses can be used to model dependent effect sizes. The advantages of the structural equation modeling approach over the multilevel approach with regard to conducting a 3-level meta-analysis are discussed. This article also seeks to extend the key concepts of Q statistics, I2, and R2 from 2-level meta-analyses to 3-level meta-analyses. The proposed procedures are implemented using the open source metaSEM package for the R statistical environment. Two real data sets are used to illustrate these procedures. New research directions related to 3-level meta-analyses are discussed.

Manifest variable path analysis: Potentially serious and misleading consequences due to uncorrected measurement error.

Abstract: Despite clear evidence that manifest variable path analysis requires highly reliable measures, path analyses with fallible measures are commonplace even in premier journals. Using fallible measures in path analysis can cause several serious problems: (a) As measurement error pervades a given data set, many path coefficients may be either over- or underestimated. (b) Extensive measurement error diminishes power and can prevent invalid models from being rejected. (c) Even a little measurement error can cause valid models to appear invalid. (d) Differential measurement error in various parts of a model can change the substantive conclusions that derive from path analysis. (e) All of these problems become increasingly serious and intractable as models become more complex. Methods to prevent and correct these problems are reviewed. The conclusion is that researchers should use more reliable measures (or correct for measurement error in the measures they do use), obtain multiple measures for use in latent variable modeling, and test simpler models containing fewer variables.

Interpretation and identification of causal mediation.

Abstract: This article reviews the foundations of causal mediation analysis and offers a general and transparent account of the conditions necessary for the identification of natural direct and indirect effects, thus facilitating a more informed judgment of the plausibility of these conditions in specific applications. I show that the conditions usually cited in the literature are overly restrictive and can be relaxed substantially without compromising identification. In particular, I show that natural effects can be identified by methods that go beyond standard adjustment for confounders, applicable to observational studies in which treatment assignment remains confounded with the mediator or with the outcome. These identification conditions can be validated algorithmically from the diagrammatic description of one's model and are guaranteed to produce unbiased results whenever the description is correct. The identi- fication conditions can be further relaxed in parametric models, possibly including interactions, and permit one to compare the relative importance of several pathways, mediated by interdependent variables. Mediation analysis aims to uncover causal pathways along which changes are transmitted from causes to effects. Interest in mediation analysis stems from both scientific and practical con- siderations. Scientifically, mediation tells us how nature works, and practically, it enables us to predict behavior under a rich variety of conditions and policy interventions. For example, in coping with the age-old problem of gender discrimination (Bickel, Hammel, & O'Connell, 1975; Goldberger, 1984), a policymaker may be interested in assessing the extent to which gender disparity in hiring can be reduced by making hiring decisions gender-blind, compared with eliminating gender inequality in education or job qualifications. The former concerns the direct effect of gender on hiring, while the latter concerns the indirect effect or the effect mediated via job qualification. The example illustrates two essential ingredients of modern mediation analysis. First, the indirect effect is not merely a mod- eling artifact formed by suggestive combinations of parameters but an intrinsic property of reality that has tangible policy implica- tions. In this example, reducing employers' prejudices and launch- ing educational reforms are two contending policy options that involve costly investments and different implementation efforts. Knowing in advance which of the two, if successful, has a greater impact on reducing hiring disparity is essential for planning and depends critically on mediation analysis for resolution. Second, the policy decisions in this example concern the enabling and dis- abling of processes (hiring vs. education) rather than lowering or raising values of specific variables. These two considerations lead to the analysis of natural direct and indirect effects. Mediation analysis has its roots in the literature of structural equation models (SEMs), going back to Wright's (1923, 1934) method of path analysis and continuing in the social sciences from the 1960s to 1980s through the works of Baron and Kenny (1986), Bollen (1989), Duncan (1975), and Fox (1980). The bulk of this work was carried out in the context of linear models, in which effect sizes are represented as sums and products of structural coefficients. The definition, identification, and estimation of these coefficients required a commitment to a particular parametric and distributional model and fell short of providing a general, causally defensible measure of mediation (Glynn, 2012; Hayes, 2009; Kraemer, Kiernan, Essex, & Kupfer, 2008; MacKinnon, 2008).

Outlier removal, sum scores, and the inflation of the Type I error rate in independent samples t tests: the power of alternatives and recommendations.

Abstract: In psychology, outliers are often excluded before running an independent samples t test, and data are often nonnormal because of the use of sum scores based on tests and questionnaires. This article concerns the handling of outliers in the context of independent samples t tests applied to nonnormal sum scores. After reviewing common practice, we present results of simulations of artificial and actual psychological data, which show that the removal of outliers based on commonly used Z value thresholds severely increases the Type I error rate. We found Type I error rates of above 20% after removing outliers with a threshold value of Z = 2 in a short and difficult test. Inflations of Type I error rates are particularly severe when researchers are given the freedom to alter threshold values of Z after having seen the effects thereof on outcomes. We recommend the use of nonparametric Mann-Whitney-Wilcoxon tests or robust Yuen-Welch tests without removing outliers. These alternatives to independent samples t tests are found to have nominal Type I error rates with a minimal loss of power when no outliers are present in the data and to have nominal Type I error rates and good power when outliers are present.

Modeling sparsely clustered data: design-based, model-based, and single-level methods.

Abstract: Recent studies have investigated the small sample properties of models for clustered data, such as multilevel models and generalized estimating equations. These studies have focused on parameter bias when the number of clusters is small, but very few studies have addressed the methods' properties with sparse data: a small number of observations within each cluster. In particular, studies have yet to address the properties of generalized estimating equations, a possible alternative to multilevel models often overlooked in behavioral sciences, with sparse data. This article begins with a discussion of population-averaged and cluster-specific models, provides a brief overview of both multilevel models and generalized estimating equations, and then conducts a simulation study on the sparse data properties of generalized estimating equations, multilevel models, and single-level regression models for both normal and binary outcomes. The simulation found generalized estimating equations estimate regression coefficients and their standard errors without bias with as few as 2 observations per cluster, provided that the number of clusters was reasonably large. Similar to the previous studies, multilevel models tended to overestimate the between-cluster variance components when the cluster size was below about 5.

Longitudinal design considerations to optimize power to detect variances and covariances among rates of change: simulation results based on actual longitudinal studies.

Abstract: We investigated the power to detect variances and covariances in rates of change in the context of existing longitudinal studies using linear bivariate growth curve models. Power was estimated by means of Monte Carlo simulations. Our findings show that typical longitudinal study designs have substantial power to detect both variances and covariances among rates of change in a variety of cognitive, physical functioning, and mental health outcomes. We performed simulations to investigate the interplay among number and spacing of occasions, total duration of the study, effect size, and error variance on power and required sample size. The relation between growth rate reliability (GRR) and effect size to the sample size required to detect power greater than or equal to .80 was nonlinear, with rapidly decreasing sample sizes needed as GRR increases. The results presented here stand in contrast to previous simulation results and recommendations (Hertzog, Lindenberger, Ghisletta, & von Oertzen, 2006; Hertzog, von Oertzen, Ghisletta, & Lindenberger, 2008; von Oertzen, Ghisletta, & Lindenberger, 2010), which are limited due to confounds between study length and number of waves, error variance with growth curve reliability, and parameter values that are largely out of bounds of actual study values. Power to detect change is generally low in the early phases (i.e., first years) of longitudinal studies but can substantially increase if the design is optimized. We recommend additional assessments, including embedded intensive measurement designs, to improve power in the early phases of long-term longitudinal studies.

Estimating Interaction Effects With Incomplete Predictor Variables

Abstract: The existing missing data literature does not provide a clear prescription for estimating interaction effects with missing data, particularly when the interaction involves a pair of continuous variables. In this article, we describe maximum likelihood and multiple imputation procedures for this common analysis problem. We outline 3 latent variable model specifications for interaction analyses with missing data. These models apply procedures from the latent variable interaction literature to analyses with a single indicator per construct (e.g., a regression analysis with scale scores). We also discuss multiple imputation for interaction effects, emphasizing an approach that applies standard imputation procedures to the product of 2 raw score predictors. We thoroughly describe the process of probing interaction effects with maximum likelihood and multiple imputation. For both missing data handling techniques, we outline centering and transformation strategies that researchers can implement in popular software packages, and we use a series of real data analyses to illustrate these methods. Finally, we use computer simulations to evaluate the performance of the proposed techniques.

Measurement and control of response styles using anchoring vignettes: a model-based approach.

Abstract: Response styles are frequently of concern when rating scales are used in psychological survey instruments. While latent trait models provide an attractive way of controlling for response style effects (Morren, Gelissen, & Vermunt, 2011), the analyses are generally limited to accommodating only a small number of response style types. The use of anchoring vignettes provides an opportunity to overcome this limitation. In this article, a new latent trait model is introduced that uses vignette responses to measure and control for any form of response style. An illustration is provided with data from a cross-national study of self-reported conscientiousness by Mottus, Allik, Realo, Pullman, et al. (2012).

Estimating causal effects from multiple-baseline studies: implications for design and analysis.

Abstract: Traditionally, average causal effects from multiple-baseline data are estimated by aggregating individual causal effect estimates obtained through within-series comparisons of treatment phase trajectories to baseline extrapolations. Concern that these estimates may be biased due to event effects, such as history and maturation, motivates our proposal of a between-series estimator that contrasts participants in the treatment to those in the baseline phase. Accuracy of the new method was assessed and compared in a series of simulation studies where participants were randomly assigned to intervention start points. The within-series estimator was found to have greater power to detect treatment effects but also to be biased due to event effects, leading to faulty causal inferences. The between-series estimator remained unbiased and controlled the Type I error rate independent of event effects. Because the between-series estimator is unbiased under different assumptions, the 2 estimates complement each other, and the difference between them can be used to detect inaccuracies in the modeling assumptions. The power to detect inaccuracies associated with event effects was found to depend on the size and type of event effect. We empirically illustrate the methods using a real data set and then discuss implications for researchers planning multiple-baseline studies.

Robust Mediation Analysis Based on Median Regression

Abstract: Mediation analysis has many applications in psychology and the social sciences The most prevalent methods typically assume that the error distribution is normal and homoscedastic However, this assumption may rarely be met in practice, which can affect the validity of the mediation analysis To address this problem, we propose robust mediation analysis based on median regression Our approach is robust to various departures from the assumption of homoscedasticity and normality, including heavy-tailed, skewed, contaminated, and heteroscedastic distributions Simulation studies show that under these circumstances, the proposed method is more efficient and powerful than standard mediation analysis We further extend the proposed robust method to multilevel mediation analysis, and demonstrate through simulation studies that the new approach outperforms the standard multilevel mediation analysis We illustrate the proposed method using data from a program designed to increase reemployment and enhance mental health of job seekers

Treatment noncompliance in randomized experiments: statistical approaches and design issues.

Abstract: Treatment noncompliance in randomized experiments threatens the validity of causal inference and the interpretability of treatment effects. This article provides a nontechnical review of 7 approaches: 3 traditional and 4 newer statistical analysis strategies. Traditional approaches include (a) intention-to-treat analysis (which estimates the effects of treatment assignment irrespective of treatment received), (b) as-treated analysis (which reassigns participants to groups reflecting the treatment they actually received), and (c) per-protocol analysis (which drops participants who did not comply with their assigned treatment). Newer approaches include (d) the complier average causal effect (which estimates the effect of treatment on the subpopulation of those who would comply with their assigned treatment), (e) dose-response estimation (which uses degree of compliance to stratify participants, producing an estimate of a dose-response relationship), (f) propensity score analysis (which uses covariates to estimate the probability that individual participants will comply, enabling estimates of treatment effects at different propensities), and (g) treatment effect bounding (which calculates a range of possible treatment effects applicable to both compliers and noncompliers). The discussion considers the areas of application, the quantity estimated, the underlying assumptions, and the strengths and weaknesses of each approach.

Modeling continuous self-report measures of perceived emotion using generalized additive mixed models.

Abstract: Emotion research has long been dominated by the "standard method" of displaying posed or acted static images of facial expressions of emotion. While this method has been useful, it is unable to investigate the dynamic nature of emotion expression. Although continuous self-report traces have enabled the measurement of dynamic expressions of emotion, a consensus has not been reached on the correct statistical techniques that permit inferences to be made with such measures. We propose generalized additive models and generalized additive mixed models as techniques that can account for the dynamic nature of such continuous measures. These models allow us to hold constant shared components of responses that are due to perceived emotion across time, while enabling inference concerning linear differences between groups. The generalized additive mixed model approach is preferred, as it can account for autocorrelation in time series data and allows emotion decoding participants to be modeled as random effects. To increase confidence in linear differences, we assess the methods that address interactions between categorical variables and dynamic changes over time. In addition, we provide comments on the use of generalized additive models to assess the effect size of shared perceived emotion and discuss sample sizes. Finally, we address additional uses, the inference of feature detection, continuous variable interactions, and measurement of ambiguity.

Bayesian evaluation of inequality constrained hypotheses

Abstract: Bayesian evaluation of inequality constrained hypotheses enables researchers to investigate their expectations with respect to the structure among model parameters. This article proposes an approximate Bayes procedure that can be used for the selection of the best of a set of inequality constrained hypotheses based on the Bayes factor in a very general class of statistical models. The software package BIG is provided such that psychologists can use the approach proposed for the analysis of their own data. To illustrate the approximate Bayes procedure and the use of BIG, we evaluate inequality constrained hypotheses in a path model and a logistic regression model. Two simulation studies on the performance of our approximate Bayes procedure show that it results in accurate Bayes factors.

Comment on Pearl: Practical implications of theoretical results for causal mediation analysis.

Abstract: Mediation analysis has been extensively applied in psychological and other social science research. A number of methodologists have recently developed a formal theoretical framework for mediation analysis from a modern causal inference perspective. In Imai, Keele, and Tingley (2010), we have offered such an approach to causal mediation analysis that formalizes identification, estimation, and sensitivity analysis in a single framework. This approach has been used by a number of substantive researchers, and in subsequent work we have also further extended it to more complex settings and developed new research designs. In an insightful article, Pearl (2014) proposed an alternative approach that is based on a set of assumptions weaker than ours. In this comment, we demonstrate that the theoretical differences between our identification assumptions and his alternative conditions are likely to be of little practical relevance in the substantive research settings faced by most psychologists and other social scientists. We also show that our proposed estimation algorithms can be easily applied in the situations discussed in Pearl (2014). The methods discussed in this comment and many more are implemented via mediation, an open-source software (Tingley, Yamamoto, Hirose, Keele, & Imai, 2013).

Coupled latent differential equation with moderators: simulation and application.

Abstract: Latent differential equations (LDE) use differential equations to analyze time series data. Because of the recent development of this technique, some issues critical to running an LDE model remain. In this article, the authors provide solutions to some of these issues and recommend a step-by-step procedure demonstrated on a set of empirical data, which models the interaction between ovarian hormone cycles and emotional eating. Results indicated that emotional eating is self-regulated. For instance, when people do more emotional eating than normal, they will subsequently tend to decrease their emotional eating behavior. In addition, a sudden increase will produce a stronger tendency to decrease than will a slow increase. We also found that emotional eating is coupled with the cycle of the ovarian hormone estradiol, and the peak of emotional eating occurs after the peak of estradiol. The self-reported average level of negative affect moderates the frequency of eating regulation and the coupling strength between eating and estradiol. Thus, people with a higher average level of negative affect tend to fluctuate faster in emotional eating, and their eating behavior is more strongly coupled with the hormone estradiol. Permutation tests on these empirical data supported the reliability of using LDE models to detect self-regulation and a coupling effect between two regulatory behaviors.

Combating unmeasured confounding in cross-sectional studies: evaluating instrumental-variable and Heckman selection models.

Abstract: Unmeasured confounding is the principal threat to unbiased estimation of treatment "effects" (i.e., regression parameters for binary regressors) in nonexperimental research. It refers to unmeasured characteristics of individuals that lead them both to be in a particular "treatment" category and to register higher or lower values than others on a response variable. In this article, I introduce readers to 2 econometric techniques designed to control the problem, with a particular emphasis on the Heckman selection model (HSM). Both techniques can be used with only cross-sectional data. Using a Monte Carlo experiment, I compare the performance of instrumental-variable regression (IVR) and HSM to that of ordinary least squares (OLS) under conditions with treatment and unmeasured confounding both present and absent. I find HSM generally to outperform IVR with respect to mean-square-error of treatment estimates, as well as power for detecting either a treatment effect or unobserved confounding. However, both HSM and IVR require a large sample to be fully effective. The use of HSM and IVR in tandem with OLS to untangle unobserved confounding bias in cross-sectional data is further demonstrated with an empirical application. Using data from the 2006-2010 General Social Survey (National Opinion Research Center, 2014), I examine the association between being married and subjective well-being.

An affirmation of the phenomenological psychological descriptive method: a response to Rennie (2012).

Abstract: Rennie (2012) made the claim that, despite their diversity, all qualitative methods are essentially hermeneutical, and he attempted to back up that claim by demonstrating that certain core steps that he called hermeneutical are contained in all of the other methods despite their self-interpretation. In this article, I demonstrate that the method I developed based upon Husserlian phenomenology cannot be so interpreted despite Rennie's effort to do so. I claim that the undertaking of a psychological investigation at large can be considered interpretive but that when the phenomenological method based upon Husserl is employed, it is descriptive. I also object to the attempt to reduce varied theoretical perspectives to the methodical steps of one of the competing theories. Reducing theoretical perspectives to core steps distorts the full value of the theoretical perspective. The last point is demonstrated by showing how the essence of the descriptive phenomenological method is missed if one follows Rennie's core steps.

The choice of product indicators in latent variable interaction models: post hoc analyses.

Abstract: The unconstrained product indicator (PI) approach is a simple and popular approach for modeling nonlinear effects among latent variables. This approach leaves the practitioner to choose the PIs to be included in the model, introducing arbitrariness into the modeling. In contrast to previous Monte Carlo studies, we evaluated the PI approach by 3 post hoc analyses applied to a real-world case adopted from a research effort in social psychology. The measurement design applied 3 and 4 indicators for the 2 latent 1st-order variables, leaving the researcher with a choice among more than 4,000 possible PI configurations. Sixty so-called matched-pair configurations that have been recommended in previous literature are of special interest. In the 1st post hoc analysis we estimated the interaction effect for all PI configurations, keeping the real-world sample fixed. The estimated interaction effect was substantially affected by the choice of PIs, also across matched-pair configurations. Subsequently, a post hoc Monte Carlo study was conducted, with varying sample sizes and data distributions. Convergence, bias, Type I error and power of the interaction test were investigated for each matched-pair configuration and the all-pairs configuration. Variation in estimates across matched-pair configurations for a typical sample was substantial. The choice of specific configuration significantly affected convergence and the interaction test's outcome. The all-pairs configuration performed overall better than the matched-pair configurations. A further advantage of the all-pairs over the matched-pairs approach is its unambiguity. The final study evaluates the all-pairs configuration for small sample sizes and compares it to the non-PI approach of latent moderated structural equations.

Functional data analysis for dynamical system identification of behavioral processes.

Abstract: Efficient new technology has made it straightforward for behavioral scientists to collect anywhere from several dozen to several thousand dense, repeated measurements on one or more time-varying variables. These intensive longitudinal data (ILD) are ideal for examining complex change over time but present new challenges that illustrate the need for more advanced analytic methods. For example, in ILD the temporal spacing of observations may be irregular, and individuals may be sampled at different times. Also, it is important to assess both how the outcome changes over time and the variation between participants' time-varying processes to make inferences about a particular intervention's effectiveness within the population of interest. The methods presented in this article integrate 2 innovative ILD analytic techniques: functional data analysis and dynamical systems modeling. An empirical application is presented using data from a smoking cessation clinical trial. Study participants provided 42 daily assessments of pre-quit and post-quit withdrawal symptoms. Regression splines were used to approximate smooth functions of craving and negative affect and to estimate the variables' derivatives for each participant. We then modeled the dynamics of nicotine craving using standard input-output dynamical systems models. These models provide a more detailed characterization of the post-quit craving process than do traditional longitudinal models, including information regarding the type, magnitude, and speed of the response to an input. The results, in conjunction with standard engineering control theory techniques, could potentially be used by tobacco researchers to develop a more effective smoking intervention.

A discrete-time Multiple Event Process Survival Mixture (MEPSUM) model.

Abstract: Traditional survival analysis was developed to investigate the occurrence and timing of a single event, but researchers have recently begun to ask questions about the order and timing of multiple events. A multiple event process survival mixture model is developed here to analyze nonrepeatable events measured in discrete-time that may occur at the same point in time. Building on both traditional univariate survival analysis and univariate survival mixture analysis, the model approximates the underlying multivariate distribution of hazard functions via a discrete-point finite mixture in which the mixing components represent prototypical patterns of event occurrence. The model is applied in an empirical analysis concerning transitions to adulthood, where the events under study include parenthood, marriage, beginning full-time work, and obtaining a college degree. Promising opportunities, as well as possible limitations of the model and future directions for research, are discussed.

Treatment effects in randomized longitudinal trials with different types of nonignorable dropout.

Abstract: Randomized longitudinal designs are commonly used in psychological and medical studies to investigate the treatment effect of an intervention or an experimental drug. Traditional linear mixed-effects models for randomized longitudinal designs are limited to maximum-likelihood methods that assume data are missing at random (MAR). In practice, because longitudinal data are often likely to be missing not at random (MNAR), the traditional mixed-effects model might lead to biased estimates of treatment effects. In such cases, an alternative approach is to utilize pattern-mixture models. In this article, a Monte Carlo simulation study compares the traditional mixed-effects model and 2 different approaches to pattern-mixture models (i.e., the differencing-averaging method and the averaging-differencing method) across different missing mechanisms (i.e., MAR, random-coefficient-dependent MNAR, or outcome-dependent MNAR) and different types of treatment-condition-based missingness. Results suggest that the traditional mixed-effects model is well suited for analyzing data with the MAR mechanism whereas the proposed pattern-mixture averaging-differencing model has the best overall performance for analyzing data with the MNAR mechanism. No method was found that could provide unbiased estimates under every missing mechanism, leading to a practical suggestion that researchers need to consider why data are missing and should also consider performing a sensitivity analysis to ascertain the extent to which their results are consistent across various missingness assumptions. Applications of different estimation methods are also illustrated using a real-data example.

Controlling the maximum familywise Type I error rate in analyses of multivariate experiments.

Abstract: Multivariate experiments are often analyzed by multistage multiple-comparison procedures (MCPs) that prohibit univariate testing on individual dependent variables if an overall multivariate analysis of variance (MANOVA) test fails to reject the relevant overall null hypothesis. Although the sole function of the MANOVA test in such analyses is to control the overall Type I error rate, it is known that the most popular MANOVA-protected MCPs do not control the maximum familywise error rate (MFWER). In this article, we show that the MFWER associated with standard MANOVA-protected MCPs can be so large that the protection provided by the initial MANOVA test is illusory. We show that the MFWER can be controlled nonconservatively with modified protected MCPs and with single-stage MCPs that allow for the construction of simultaneous confidence intervals on effect sizes. We argue that, given the ease with which these MCPs can be implemented, there is no justification for continued use of the standard procedures.

Reply to Commentary by Imai, Keele, Tingley, and Yamamoto, concerning Causal Mediation Analysis

Abstract: Forthcoming: Psychological Methods (2014) with discussion of Interpretation and Identification of Causal Mediation, R-389. TECHNICAL REPORT R-421 November 20132 Reply to Commentary by Imai, Keele, Tingley, and Yamamoto, concerning Causal Mediation Analysis Judea Pearl ∗ Computer Science Department University of California, Los Angeles Los Angeles, CA, 90095-1596 judea@cs.ucla.edu (310) 825-3243 Tel / (310) 794-5057 Fax November 18, 2013 I am happy to join Imai, Keele, Tingley, and Yammamoto (henceforth Imai-et al.) in celebrating the full convergence of our respective analyses towards a unified understanding of causal mediation. I am referring to the analysis presented in (Pearl, 2001) (reproduced in (Pearl, 2013)) on the one hand, and the analyses and implementations of (Imai et al., 2010a,b,c), on the other. In fact, when I first read (Imai et al., 2010c), I had no doubt that, despite some dissimilarities in the presentation of the assumptions, the two works would coincide on all fronts: Definitions, basic assumptions, identification and estimation algorithms. The reasons for my confidence was that, in 2001, I approached the mediation problem from the symbiotic mathematical framework of Structural Causal Models (SCM) (Pearl, 2000, Chapter 7) which unifies the graphical, potential outcomes and structural equation frameworks, and according to which, the latter two are logical equivalent; a theorem in one is a theorem in the other. They differ only in the language in which assumptions are cast. This means that even researchers who accept no other interpretation of causation except the one dictated by orthodox potential outcomes can safely use the transparency and inferential power provided by the symbiotic framework, and be assured the validity of the results. Inspired by this assurance, I derived identification conditions in the algebra of counterfactuals and presented them in two languages, counterfactual (or potential outcomes) and graphical. Not surprisingly, the mediation formulas derived in Imai et al. (2010c) coincide precisely with those derived in Pearl (2001, Eqs. (8), (17), (26), (27)). This is to be expected, since the two are but variants of the same mathematical umbrella, differing merely in the type of assumptions one is willing to posit and defend, and the language one chooses to communicate the assumptions. This commentary has benefited from discussions with Kosuke Imai, David Kenny, and Bengt Muth´en. I am grateful to Associate Editor, Patrick Shrout for giving me the opportunity to reply to this commentary. This research was supported in parts by grants from NIH #1R01 LM009961-01, NSF #IIS-0914211 and #IIS-1018922, and ONR #N000-14-09-1-0665 and #N00014-10-1-0933.

Converting from d to r to z when the design uses extreme groups, dichotomization, or experimental control.

Abstract: Meta-analyses of the relationship between 2 continuous variables sometimes involves conversions between different effect sizes, but methodological literature offers conflicting guidance about how to make such conversions. This article provides methods for converting from a standardized mean difference to a correlation coefficient (and from there to Fisher's z) under 3 types of study designs: extreme groups, dichotomization of a continuous variable, and controlled experiments. Also provided are formulas and recommendations regarding how the sampling variance of effect size statistics should be estimated in each of these cases. The conversion formula for extreme groups designs, originally due to Feldt (1961), can be viewed as a generalization of Hunter and Schmidt's (1990) method for dichotomization designs. A simulation study examines the finite-sample properties of the proposed methods. The conclusion highlights areas where current guidance in the literature should be amended or clarified.

Estimating the prevalence of socially sensitive behaviors: attributing guilty and innocent noncompliance with the single sample count method.

Abstract: Prevalence estimation models, using randomized or fuzzy responses, provide protection against exposure to respondents beyond anonymity and represent a useful research tool in socially sensitive situations. However, both guilty and innocent noncompliance can have a profound impact on prevalence estimations derived from these models. In this article, we introduce the maximum-likelihood extension of the single sample count (SSC-MLE) estimation model to detect and attribute noncompliance through testing 5 competing hypotheses on possible ways of noncompliance. We demonstrate the ability of the SSC-MLE to estimate and attribute noncompliance with a single sample using the observed distribution of affirmative answers on recent recreational drug use from a sample of university students (N = 1,441). Based on the survey answers, the drug use prevalence was estimated at 17.62% (� 6.75%), which is in line with relevant drug use statistics. Only 2.51% (� 1.54%) were noncompliant, of which 0.55% (� 0.44%) was attributed to guilty noncompliance (i.e., have used drugs but did not admit) and 2.17% (� 1.44%) to innocent noncompliers with no drug use in the past 3 months to hide. The SSC-MLE indirect estimation method represents an important tool for estimating the prevalence of a broad range of socially sensitive behaviors. Subsequent applications of the SSC-MLE to a range of transgressive behaviors with varying sensitivity will contribute to establishing the SSC-MLE's performance properties, along with obtaining empirical evidence to test the underlying assumption of independence of noncompliance from involvement. Freely downloadable, user-friendly software to facilitate applications of the SSC-MLE model is provided.

How Bandwidth Selection Algorithms Impact Exploratory Data Analysis Using Kernel Density Estimation

Abstract: Exploratory data analysis (EDA) can reveal important features of underlying distributions, and these features often have an impact on inferences and conclusions drawn from data Graphical analysis is central to EDA, and graphical representations of distributions often benefit from smoothing A viable method of estimating and graphing the underlying density in EDA is kernel density estimation (KDE) This article provides an introduction to KDE and examines alternative methods for specifying the smoothing bandwidth in terms of their ability to recover the true density We also illustrate the comparison and use of KDE methods with 2 empirical examples Simulations were carried out in which we compared 8 bandwidth selection methods (Sheather-Jones plug-in [SJDP], normal rule of thumb, Silverman's rule of thumb, least squares cross-validation, biased cross-validation, and 3 adaptive kernel estimators) using 5 true density shapes (standard normal, positively skewed, bimodal, skewed bimodal, and standard lognormal) and 9 sample sizes (15, 25, 50, 75, 100, 250, 500, 1,000, 2,000) Results indicate that, overall, SJDP outperformed all methods However, for smaller sample sizes (25 to 100) either biased cross-validation or Silverman's rule of thumb was recommended, and for larger sample sizes the adaptive kernel estimator with SJDP was recommended Information is provided about implementing the recommendations in the R computing language

Switching principal component analysis for modeling means and covariance changes over time.

Abstract: Many psychological theories predict that cognitions, affect, action tendencies, and other variables change across time in mean level as well as in covariance structure. Often such changes are rather abrupt, because they are caused by sudden events. To capture such changes, one may repeatedly measure the variables under study for a single individual and examine whether the resulting multivariate time series contains a number of phases with different means and covariance structures. The latter task is challenging, however. First, in many cases, it is unknown how many phases there are and when new phases start. Second, often a rather large number of variables is involved, complicating the interpretation of the covariance pattern within each phase. To take up this challenge, we present switching principal component analysis (PCA). Switching PCA detects phases of consecutive observations or time points (in single subject data) with similar means and/or covariation structures, and performs a PCA per phase to yield insight into its covariance structure. An algorithm for fitting switching PCA solutions as well as a model selection procedure are presented and evaluated in a simulation study. Finally, we analyze empirical data on cardiorespiratory recordings.

Practical Implications of Theoretical Results for Causal Mediation Analysis

Abstract: Mediation analysis has been extensively applied in psychological and other social science research. A number of methodologists have recently developed a formal theoretical framework for mediation analysis from a modern causal inference perspective. In Imai, Keele, and Tingley (2010), we have offered such an approach to causal mediation analysis that formalizes identification, estimation, and sensitivity analysis in a single framework. This approach has been used by a number of substantive researchers, and in subsequent work we have also further extended it to more complex settings and developed new research designs. In an insightful article, Pearl (2014) proposed an alternative approach that is based on a set of assumptions weaker than ours. In this comment, we demonstrate that the theoretical differences between our identification assumptions and his alternative conditions are likely to be of little practical relevance in the substantive research settings faced by most psychologists and other social scientists. We also show that our proposed estimation algorithms can be easily applied in the situations discussed in Pearl (2014). The methods discussed in this comment and many more are implemented via mediation, an open-source software (Tingley, Yamamoto, Hirose, Keele, & Imai, 2013).

Cognitive Diagnostic Assessment via Bayesian Evaluation of Informative Diagnostic Hypotheses

Abstract: There exist diverse approaches that can be used for cognitive diagnostic assessment, such as mastery testing, constrained latent class analysis, rule space methodology, diagnostic cognitive modeling, and person-fit analysis. Each of these approaches can be used within 1 of the 4 psychometric perspectives on diagnostic testing discussed by Borsboom (2008), that is, the dimensional, diagnostic, constructivist, and causal system perspectives. Bayesian evaluation of informative diagnostic hypotheses is an alternative for each of the other approaches that is more flexible in the diagnostic hypotheses that can be evaluated, and it can be used in each of the 4 psychometric perspectives on diagnostic testing. After being formulated, informative diagnostic hypotheses are evaluated by means of the Bayes factor using only the data from the person to be diagnosed. Already, relatively small diagnostic tests render Bayes factors that provide convincing evidence in favor of 1 of the diagnostic hypotheses under consideration