Cryptic multiple hypotheses testing in linear models: overestimated effect sizes and the winner's curse.
Reads0
Chats0
TLDR
Full model tests and P value adjustments can be used as a guide to how frequently type I errors arise by sampling variation alone, and favour the presentation of full models, since they best reflect the range of predictors investigated and ensure a balanced representation also of non-significant results.Abstract:
Fitting generalised linear models (GLMs) with more than one predictor has become the standard method of analysis in evolutionary and behavioural research. Often, GLMs are used for exploratory data analysis, where one starts with a complex full model including interaction terms and then simplifies by removing non-significant terms. While this approach can be useful, it is problematic if significant effects are interpreted as if they arose from a single a priori hypothesis test. This is because model selection involves cryptic multiple hypothesis testing, a fact that has only rarely been acknowledged or quantified. We show that the probability of finding at least one ‘significant’ effect is high, even if all null hypotheses are true (e.g. 40% when starting with four predictors and their two-way interactions). This probability is close to theoretical expectations when the sample size (N) is large relative to the number of predictors including interactions (k). In contrast, type I error rates strongly exceed even those expectations when model simplification is applied to models that are over-fitted before simplification (low N/k ratio). The increase in false-positive results arises primarily from an overestimation of effect sizes among significant predictors, leading to upward-biased effect sizes that often cannot be reproduced in follow-up studies (‘the winner's curse’). Despite having their own problems, full model tests and P value adjustments can be used as a guide to how frequently type I errors arise by sampling variation alone. We favour the presentation of full models, since they best reflect the range of predictors investigated and ensure a balanced representation also of non-significant results.read more
Citations
More filters
Journal ArticleDOI
OUP accepted manuscript
TL;DR: In this article , the authors experimentally tested the use of acoustic cues in wild zebra finches (Taeniopygia guttata) as they flew around their arid habitat as a way of locating conspecifics on the ground and potentially accessing useful social information.
Journal ArticleDOI
Preschoolers agree to and enforce prosocial, but not selfish, sharing norms
TL;DR: The authors used a group dictator game to investigate whether 3-and 5-year-olds would agree to and enforce a selfish or prosocial sharing norm in a resource sharing context, and they found that older preschoolers also enforced the prosocial norm by protesting normatively and intervening when the protagonist acted selfishly (and thus violated the norm).
Journal ArticleDOI
Drivers of gut microbiome variation within and between groups of a wild Malagasy primate
Katja Rudolph,Dominik Schneider,Claudia Fichtel,Rolf Daniel,Michael Heistermann,Peter M. Kappeler +5 more
TL;DR: This paper examined the potential drivers of the convergence of the gut microbiome on multiple scales among and within seven neighbouring groups of wild Verreaux's sifakas (Propithecus verreauxi) -a folivorous primate of Madagascar.
Journal ArticleDOI
Difference in plasticity of resting metabolic rate - the proximate explanation to different niche breadth in sympatric Ficedula flycatchers.
TL;DR: It is suggested that a more flexible nestling RMR in piedFlycatchers is responsible for the better adaptation of pied flycatchers to the typical seasonal changes in food availability experienced in this hybrid zone.
Journal ArticleDOI
Intra-specific variation in nest-site preferences of Zebra Finches: do height and cover matter?
TL;DR: Investigation of nest-height and nest-cover preferences of five Zebra Finch populations revealed that all five populations preferred the highest nests and there were no differences between populations, and discussed the intra-specific variation in nest-site preferences, in the light of evolutionary and domestication processes.
References
More filters
Journal ArticleDOI
Controlling the false discovery rate: a practical and powerful approach to multiple testing
Yoav Benjamini,Yosef Hochberg +1 more
TL;DR: In this paper, a different approach to problems of multiple significance testing is presented, which calls for controlling the expected proportion of falsely rejected hypotheses -the false discovery rate, which is equivalent to the FWER when all hypotheses are true but is smaller otherwise.
Book
Model Selection and Multimodel Inference: A Practical Information-Theoretic Approach
TL;DR: The second edition of this book is unique in that it focuses on methods for making formal statistical inference from all the models in an a priori set (Multi-Model Inference).
Book
Multiple Regression: Testing and Interpreting Interactions
Leona S. Aiken,Stephen G. West +1 more
TL;DR: In this article, the effects of predictor scaling on the coefficients of regression equations are investigated. But, they focus mainly on the effect of predictors scaling on coefficients of regressions.
Book
Discovering Statistics Using SPSS
Andy P. Field,Jeremy N.V. Miles +1 more
TL;DR: Suitable for those new to statistics as well as students on intermediate and more advanced courses, the book walks students through from basic to advanced level concepts, all the while reinforcing knowledge through the use of SAS(R).
Journal ArticleDOI
A Simple Sequentially Rejective Multiple Test Procedure
TL;DR: In this paper, a simple and widely accepted multiple test procedure of the sequentially rejective type is presented, i.e. hypotheses are rejected one at a time until no further rejections can be done.