Why we (usually) don't have to worry about multiple comparisons
TLDR
In this article, the problem of multiple comparisons can disappear entirely when viewed from a hierarchical Bayesian perspective, and a multilevel model is proposed to address the multiple comparisons problem and also yield more efficient estimates.Abstract:
Applied researchers often find themselves making statistical inferences in settings that would seem to require multiple comparisons adjustments. We challenge the Type I error paradigm that underlies these corrections. Moreover we posit that the problem of multiple comparisons can disappear entirely when viewed from a hierarchical Bayesian perspective. We propose building multilevel models in the settings where multiple comparisons arise. Multilevel models perform partial pooling (shifting estimates toward each other), whereas classical procedures typically keep the centers of intervals stationary, adjusting for multiple comparisons by making the intervals wider (or, equivalently, adjusting the p values corresponding to intervals of fixed width). Thus, multilevel models address the multiple comparisons problem and also yield more efficient estimates, especially in settings with low group-level variation, which is where multiple comparisons are a particular concern.read more
Citations
More filters
Journal ArticleDOI
Bayesian data analysis.
TL;DR: A fatal flaw of NHST is reviewed and some benefits of Bayesian data analysis are introduced and illustrative examples of multiple comparisons in Bayesian analysis of variance and Bayesian approaches to statistical power are presented.
Journal ArticleDOI
Stan : A Probabilistic Programming Language
Bob Carpenter,Andrew Gelman,Matthew D. Hoffman,Daniel D. Lee,Ben Goodrich,Michael Betancourt,Marcus A. Brubaker,Jiqiang Guo,Peter Li,Allen Riddell +9 more
TL;DR: Stan as discussed by the authors is a probabilistic programming language for specifying statistical models, where a program imperatively defines a log probability function over parameters conditioned on specified data and constants, which can be used in alternative algorithms such as variational Bayes, expectation propagation, and marginal inference using approximate integration.
Stan: A Probabilistic Programming Language.
Bob Carpenter,Andrew Gelman,Matthew D. Hoffman,Daniel D. Lee,Ben Goodrich,Michael Betancourt,Marcus A. Brubaker,Jiqiang Guo,Peter Li,Allen Riddell +9 more
TL;DR: Stan is a probabilistic programming language for specifying statistical models that provides full Bayesian inference for continuous-variable models through Markov chain Monte Carlo methods such as the No-U-Turn sampler and an adaptive form of Hamiltonian Monte Carlo sampling.
Journal ArticleDOI
Bayesian Estimation Supersedes the t Test
TL;DR: Bayesian estimation for 2 groups provides complete distributions of credible values for the effect size, group means and their difference, standard deviations and their Difference, and the normality of the data.
Journal ArticleDOI
Structural topic models for open ended survey responses
Margaret E. Roberts,Brandon M. Stewart,Dustin Tingley,Chris Lucas,Jetson Leder-Luis,Shana Kushner Gadarian,Bethany Albertson,David G. Rand +7 more
TL;DR: The structural topic model makes analyzing open-ended responses easier, more revealing, and capable of being used to estimate treatment effects, and is illustrated with analysis of text from surveys and experiments.
References
More filters
Journal ArticleDOI
Controlling the false discovery rate: a practical and powerful approach to multiple testing
Yoav Benjamini,Yosef Hochberg +1 more
TL;DR: In this paper, a different approach to problems of multiple significance testing is presented, which calls for controlling the expected proportion of falsely rejected hypotheses -the false discovery rate, which is equivalent to the FWER when all hypotheses are true but is smaller otherwise.
Book
Bayesian Data Analysis
TL;DR: Detailed notes on Bayesian Computation Basics of Markov Chain Simulation, Regression Models, and Asymptotic Theorems are provided.
Journal Article
Statistical Comparisons of Classifiers over Multiple Data Sets
TL;DR: A set of simple, yet safe and robust non-parametric tests for statistical comparisons of classifiers is recommended: the Wilcoxon signed ranks test for comparison of two classifiers and the Friedman test with the corresponding post-hoc tests for comparisons of more classifiers over multiple data sets.
Journal ArticleDOI
The control of the false discovery rate in multiple testing under dependency
Yoav Benjamini,Daniel Yekutieli +1 more
TL;DR: In this paper, it was shown that a simple FDR controlling procedure for independent test statistics can also control the false discovery rate when test statistics have positive regression dependency on each of the test statistics corresponding to the true null hypotheses.
Book
Data Analysis Using Regression and Multilevel/Hierarchical Models
Andrew Gelman,Yu-Sung Su +1 more
TL;DR: Data Analysis Using Regression and Multilevel/Hierarchical Models is a comprehensive manual for the applied researcher who wants to perform data analysis using linear and nonlinear regression and multilevel models.