Showing papers by "Donald B. Rubin published in 2016"

Asymptotic Theory of Rerandomization in Treatment-Control Experiments

Abstract: Although complete randomization ensures covariate balance on average, the chance for observing significant differences between treatment and control covariate distributions increases with many covariates. Rerandomization discards randomizations that do not satisfy a predetermined covariate balance criterion, generally resulting in better covariate balance and more precise estimates of causal effects. Previous theory has derived finite sample theory for rerandomization under the assumptions of equal treatment group sizes, Gaussian covariate and outcome distributions, or additive causal effects, but not for the general sampling distribution of the difference-in-means estimator for the average causal effect. To supplement existing results, we develop asymptotic theory for rerandomization without these assumptions, which reveals a non-Gaussian asymptotic distribution for this estimator, specifically a linear combination of a Gaussian random variable and a truncated Gaussian random variable. This distribution follows because rerandomization affects only the projection of potential outcomes onto the covariate space but does not affect the corresponding orthogonal residuals. We also demonstrate that, compared to complete randomization, rerandomization reduces the asymptotic sampling variances and quantile ranges of the difference-in-means estimator. Moreover, our work allows the construction of accurate large-sample confidence intervals for the average causal effect, thereby revealing further advantages of rerandomization over complete randomization.

Improving covariate balance in 2K factorial designs via rerandomization with an application to a New York City Department of Education High School Study

Abstract: A few years ago, the New York Department of Education (NYDE) was planning to conduct an experiment involving five new intervention programs for a selected set of New York City high schools. The goal was to estimate the causal effects of these programs and their interactions on the schools’ performance. For each of the schools, about 50 premeasured covariates were available. The schools could be randomly assigned to the 32 treatment combinations of this $2^{5}$ factorial experiment, but such an allocation could have resulted in a huge covariate imbalance across treatment groups. Standard methods used to prevent confounding of treatment effects with covariate effects (e.g., blocking) were not intuitive due to the large number of covariates. In this paper, we explore how the recently proposed and studied method of rerandomization can be applied to this problem and other factorial experiments. We propose how to implement rerandomization in factorial experiments, extend the theoretical properties of rerandomization from single-factor experiments to $2^{K}$ factorial designs, and demonstrate, using the NYDE data, how such a designed experiment can improve precision of estimated factorial effects.

A Bayesian Perspective on the Analysis of Unreplicated Factorial Experiments Using Potential Outcomes

Abstract: Unreplicated factorial designs have been widely used in scientific and industrial settings, when it is important to distinguish “active” or real factorial effects from “inactive” or noise factorial effects used to estimate residual or “error” terms. We propose a new approach to screen for active factorial effects from such experiments that uses the potential outcomes framework and is based on sequential posterior predictive model checks. One advantage of the proposed method is its ability to broaden the standard definition of active effects and to link their definition to the population of interest. Another important aspect of this approach is its conceptual connection to Fisherian randomization tests. Extensive simulation studies are conducted, which demonstrate the superiority of the proposed approach over existing ones in the situations considered.

Sample size determination using posterior predictive distributions

Abstract: SUMMARY. A statistical model developed from scientific theory may "fail to fit" the available data if the scientific theory is incorrect or if the sample size is too small. The former point is obvious but the latter is more subtle. In the latter case, the hypothesized model may fail to fit in the sense that it is viewed as unnecessarily complicated, and so the investigators settle upon a simpler model that ignores structure hypothesized by scientific theory. We describe a simulation-based approach for determining the sample size that would be required for distinguishing between the simpler model and the hypothesized model assuming the latter is correct. Data are simulated assuming the hypothesized model is correct and compared to posterior predictive replications of the data, which are drawn assuming the simpler model is correct. This is repeated for a number of sample sizes. The Bayesian approach offers two especially nice features for addressing a problem of this type: first, we can average over a variety of plausible values for the parameters of the hypothesized model rather than fixing a single alternative; second, the approach does not require that we restrict attention to a limited class of regular models (e.g., t-tests or linear models). The posterior predictive approach to sample size determination is illustrated using an application of finite mixture models to psychological data.

The fragility of standard inferential approaches in principal stratification models relative to direct likelihood approaches

Abstract: Many empirical settings involve the specification of models leading to complicated likelihood functions, for example, finite mixture models that arise in causal inference when using Principal Stratification PS. Traditional asymptotic results cannot be trusted for the associated likelihood functions, whose logarithms are not close to being quadratic and may be multimodal even with large sample sizes. We first investigate the shape of the likelihood function with models based on PS by providing diagnostic tools for evaluating ellipsoidal approximations based on the second derivatives of the log-likelihood at a mode. In these settings, inference based on standard approximations is inappropriate, and other forms of inference are required. We explore the use of a direct likelihood approach for parsimonious model selection and, specifically, propose comparing values of scaled maximized likelihood functions under competitive models to select preferred models. An extensive simulation study provides guidelines, for calibrating the use of scaled log-likelihood ratio statistics, as functions of the complexity of the models being compared.

Non-standard conditionally specified models for non-ignorable missing data

Abstract: Data analyses typically rely upon assumptions about missingness mechanisms that lead to observed versus missing data. When the data are missing not at random, direct assumptions about the missingness mechanism, and indirect assumptions about the distributions of observed and missing data, are typically untestable. We explore an approach, where the joint distribution of observed data and missing data is specified through non-standard conditional distributions. In this formulation, which traces back to a factorization of the joint distribution, apparently proposed by J.W. Tukey, the modeling assumptions about the conditional factors are either testable or are designed to allow the incorporation of substantive knowledge about the problem at hand, thereby offering a possibly realistic portrayal of the data, both missing and observed. We apply Tukey's conditional representation to exponential family models, and we propose a computationally tractable inferential strategy for this class of models. We illustrate the utility of this approach using high-throughput biological data with missing data that are not missing at random.

Causal Inference in Rebuilding and Extending the Recondite Bridge between Finite Population Sampling and Experimental Design

Abstract: This article considers causal inference for treatment contrasts from a randomized experiment using potential outcomes in a finite population setting. Adopting a Neymanian repeated sampling approach that integrates such causal inference with finite population survey sampling, an inferential framework is developed for general mechanisms of assigning experimental units to multiple treatments. This framework extends classical methods by allowing the possibility of randomization restrictions and unequal replications. Novel conditions that are "milder" than strict additivity of treatment effects, yet permit unbiased estimation of the finite population sampling variance of any treatment contrast estimator, are derived. The consequences of departures from such conditions are also studied under the criterion of minimax bias, and a new justification for using the Neymanian conservative sampling variance estimator in experiments is provided. The proposed approach can readily be extended to the case of treatments with a general factorial structure.

Fisher, Neyman, and Bayes at FDA.

Abstract: The wise use of statistical ideas in practice essentially requires some Bayesian thinking, in contrast to the classical rigid frequentist dogma. This dogma too often has seemed to influence the applications of statistics, even at agencies like the FDA. Greg Campbell was one of the most important advocates there for more nuanced modes of thought, especially Bayesian statistics. Because two brilliant statisticians, Ronald Fisher and Jerzy Neyman, are often credited with instilling the traditional frequentist approach in current practice, I argue that both men were actually seeking very Bayesian answers, and neither would have endorsed the rigid application of their ideas.

Evaluating the Validity of Post-Hoc Subgroup Inferences: A Case Study

Abstract: In randomized experiments, the random assignment of units to treatment groups justifies many of the widely used traditional analysis methods for evaluating causal effects. Specifying subgroups of units for further examination after observing outcomes, however, may partially nullify any advantages of randomized assignment when data are analyzed naively. Some previous statistical literature has treated all post-hoc subgroup analyses homogeneously as entirely invalid and thus uninterpretable. The extent of the validity of such analyses and the factors that affect the degree of validity remain largely unstudied. Here, we describe a recent pharmaceutical case with First Amendment legal implications, in which post-hoc subgroup analyses played a pivotal and controversial role. Through Monte Carlo simulation, we show that post-hoc results that seem highly significant make dramatic movements toward insignificance after accounting for the subgrouping procedure presumably used. Finally, we propose a novel, randomiza...