When to adjust alpha during multiple testing: A consideration of disjunction, conjunction, and individual testing
Reads0
Chats0
TLDR
This article argued that alpha adjustment is only appropriate in the case of disjunction testing, in which at least one test result must be significant in order to reject the associated joint null hypothesis.Abstract:
Scientists often adjust their significance threshold (alpha level) during null hypothesis significance testing in order to take into account multiple testing and multiple comparisons. This alpha adjustment has become particularly relevant in the context of the replication crisis in science. The present article considers the conditions in which this alpha adjustment is appropriate and the conditions in which it is inappropriate. A distinction is drawn between three types of multiple testing: disjunction testing, conjunction testing, and individual testing. It is argued that alpha adjustment is only appropriate in the case of disjunction testing, in which at least one test result must be significant in order to reject the associated joint null hypothesis. Alpha adjustment is inappropriate in the case of conjunction testing, in which all relevant results must be significant in order to reject the joint null hypothesis. Alpha adjustment is also inappropriate in the case of individual testing, in which each individual result must be significant in order to reject each associated individual null hypothesis. The conditions under which each of these three types of multiple testing is warranted are examined. It is concluded that researchers should not automatically (mindlessly) assume that alpha adjustment is necessary during multiple testing. Illustrations are provided in relation to joint studywise hypotheses and joint multiway ANOVAwise hypotheses.read more
Citations
More filters
Journal ArticleDOI
Exploratory hypothesis tests can be more compelling than confirmatory hypothesis tests
Mark J. Rubin,Chris Donkin +1 more
TL;DR: For example, this paper argued that exploratory hypothesis tests avoid researcher commitment and researcher prophecy biases, reduce the probability of data fraud, facilitate inference to the best explanation, and allow peer reviewers to make additional contributions at the data analysis stage.
Journal ArticleDOI
Understanding language preference: Autism knowledge, experience of stigma and autism identity.
Simon M. Bury,Rachel Jellett,Alexandra Haschek,Michael Wenzel,Darren Hedley,Jennifer R. Spoor +5 more
TL;DR: For instance, the authors found that those who endorse a stronger autism identity tended to find identity-first terms more preferable and less offensive, whereas those who reported greater experiences and internalisation of stigma found identity first terms less preferable and more offensive.
Journal ArticleDOI
Divergence versus decision P‐values: A distinction worth making in theory and keeping in practice: Or, how divergence P‐values measure evidence even when decision P‐values do not
TL;DR: In this paper , it is argued that divergence and decision P•values should be carefully distinguished in teaching, and that divergence P'values are the relevant choice when the analysis goal is to summarize evidence rather than implement a decision rule.
Journal ArticleDOI
Neural heterogeneity underlying late adolescent motivational processing is linked to individual differences in behavioral sensation seeking
TL;DR: In this paper , a person-specific approach to the study of adolescent functional connectivity during a continuous motivational state, and to examine links between connectivity and self-reported sensation seeking behavior in 104 adolescents (MAge = 19.3; SDAge = 1.3).
Journal ArticleDOI
Something distressing this way comes: The effects of trigger warnings on avoidance behaviors in an analogue trauma task
TL;DR: This paper found that participants did not overall avoid negative stimuli prefaced with a trigger warning any more than stimuli without a warning, and only 12.56% (n = 25) participants used the option to cover such images when given the opportunity to do so.
References
More filters
Journal ArticleDOI
Controlling the false discovery rate: a practical and powerful approach to multiple testing
Yoav Benjamini,Yosef Hochberg +1 more
TL;DR: In this paper, a different approach to problems of multiple significance testing is presented, which calls for controlling the expected proportion of falsely rejected hypotheses -the false discovery rate, which is equivalent to the FWER when all hypotheses are true but is smaller otherwise.
Journal ArticleDOI
What's wrong with Bonferroni adjustments
TL;DR: This paper advances the view, widely held by epidemiologists, that Bonferroni adjustments are, at best, unnecessary and, at worst, deleterious to sound statistical inference.
Journal ArticleDOI
No adjustments are needed for multiple comparisons.
TL;DR: A policy of not making adjustments for multiple comparisons is preferable because it will lead to fewer errors of interpretation when the data under evaluation are not random numbers but actual observations on nature.
Book
Simultaneous Statistical Inference
TL;DR: In this article, the authors presented a case of two means regression method for the family error rate, which was used to estimate the probability of a family having a nonzero family error.
Journal ArticleDOI
Rectangular Confidence Regions for the Means of Multivariate Normal Distributions
TL;DR: For rectangular confidence regions for the mean values of multivariate normal distributions, this paper proved that a confidence region constructed for independent coordinates is, at the same time, a conservative confidence region for any case of dependent coordinates.
Related Papers (5)
Manipulating the Alpha Level Cannot Cure Significance Testing
David Trafimow,Valentin Amrhein,Valentin Amrhein,Corson N. Areshenkoff,Carlos Barrera-Causil,Eric J. Beh,Yusuf K. Bilgic,Roser Bono,M. T. Bradley,William M. Briggs,Héctor A. Cepeda-Freyre,Sergio E. Chaigneau,Daniel R. Ciocca,Juan Carlos Correa,Denis Cousineau,Michiel R. de Boer,Subhra Sankar Dhar,Igor Dolgov,Juana Gómez-Benito,Marian Grendar,Marian Grendar,James W. Grice,Martin E. Guerrero-Gimenez,Andrés Gutiérrez,Tania B. Huedo-Medina,Klaus Jaffe,Armina Janyan,Ali Karimnezhad,Fränzi Korner-Nievergelt,Koji Kosugi,Martin Lachmair,Rubén Daniel Ledesma,Roberto Limongi,Marco Tullio Liuzza,Rosaria Lombardo,Michael J. Marks,Gunther Meinlschmidt,Gunther Meinlschmidt,Ladislas Nalborczyk,Ladislas Nalborczyk,Hung T. Nguyen,Raydonal Ospina,Jose D Perezgonzalez,Roland Pfister,Juan José Rahona,David A. Rodriguez-Medina,Xavier Romão,Susana Ruiz-Fernández,Susana Ruiz-Fernández,Susana Ruiz-Fernández,Isabel Suarez,Marion Tegethoff,Mauricio Tejo,Rens van de Schoot,Rens van de Schoot,Ivan Vankov,Santiago Velasco-Forero,Tonghui Wang,Yuki Yamada,Felipe Carlos Martín Zoppino,Fernando Marmolejo-Ramos +60 more