Cryptic multiple hypotheses testing in linear models: overestimated effect sizes and the winner's curse.
Reads0
Chats0
TLDR
Full model tests and P value adjustments can be used as a guide to how frequently type I errors arise by sampling variation alone, and favour the presentation of full models, since they best reflect the range of predictors investigated and ensure a balanced representation also of non-significant results.Abstract:
Fitting generalised linear models (GLMs) with more than one predictor has become the standard method of analysis in evolutionary and behavioural research. Often, GLMs are used for exploratory data analysis, where one starts with a complex full model including interaction terms and then simplifies by removing non-significant terms. While this approach can be useful, it is problematic if significant effects are interpreted as if they arose from a single a priori hypothesis test. This is because model selection involves cryptic multiple hypothesis testing, a fact that has only rarely been acknowledged or quantified. We show that the probability of finding at least one ‘significant’ effect is high, even if all null hypotheses are true (e.g. 40% when starting with four predictors and their two-way interactions). This probability is close to theoretical expectations when the sample size (N) is large relative to the number of predictors including interactions (k). In contrast, type I error rates strongly exceed even those expectations when model simplification is applied to models that are over-fitted before simplification (low N/k ratio). The increase in false-positive results arises primarily from an overestimation of effect sizes among significant predictors, leading to upward-biased effect sizes that often cannot be reproduced in follow-up studies (‘the winner's curse’). Despite having their own problems, full model tests and P value adjustments can be used as a guide to how frequently type I errors arise by sampling variation alone. We favour the presentation of full models, since they best reflect the range of predictors investigated and ensure a balanced representation also of non-significant results.read more
Citations
More filters
Journal ArticleDOI
Underappreciated problems of low replication in ecological field studies.
Nathan P. Lemoine,Ava M. Hoffman,Andrew J. Felton,Lauren E. Baur,Francis A. Chaves,Jesse Gray,Qiang Yu,Melinda D. Smith +7 more
TL;DR: A meta-analysis is conducted to determine the average statistical power and Type M error rate for manipulative field experiments that address important questions related to global change; global warming, biodiversity loss, and drought.
Journal ArticleDOI
Assessment of identity disequilibrium and its relation to empirical heterozygosity fitness correlations: a meta-analysis.
TL;DR: It is found that the magnitude of g2 was associated with the average effect sizes observed in a population, even when point estimates were nonsignificant, and suggest that many more markers than typically used are needed to robustly detect HFCs.
Journal ArticleDOI
Hunting, Law Enforcement, and African Primate Conservation
Paul K. N'Goran,Christophe Boesch,Roger Mundry,Eliézer K. N’Goran,Ilka Herbinger,Fabrice Ayé Yapi,Hjalmar S. Kühl +6 more
TL;DR: The effects of hunting on monkeys varied among species, and Red colobus monkeys were most affected and Campbell's monkeys were least affected, while density of monkeys irrespective of species was up to 100 times higher near a research station and tourism site in the southwestern section of the park, where there is little hunting, than in the southeastern part of the Park.
Journal ArticleDOI
Human local adaptation of the TRPM8 cold receptor along a latitudinal cline.
Felix M. Key,Muslihudeen A. Abdul-Aziz,Roger Mundry,Benjamin M. Peter,Aarthi Sekar,Mauro D Amato,Megan Y. Dennis,Joshua M. Schmidt,Aida M. Andrés,Aida M. Andrés +9 more
TL;DR: Focusing on cold perception–which is central to thermoregulation and survival in cold environments–it is hypothesized that local adaptation on previously neutral standing variation may have contributed to the genetic differences that exist in the prevalence of migraine among human populations today.
Journal ArticleDOI
Tonic immobility is a measure of boldness toward predators: an application of Bayesian structural equation modeling
TL;DR: Individual variation in TI in a wild vertebrate can be interpreted in a context of boldness toward predators, making TI a meaningful and practical behavioral trait for studies involving personality and antipredation behavior in wild populations.
References
More filters
Journal ArticleDOI
Controlling the false discovery rate: a practical and powerful approach to multiple testing
Yoav Benjamini,Yosef Hochberg +1 more
TL;DR: In this paper, a different approach to problems of multiple significance testing is presented, which calls for controlling the expected proportion of falsely rejected hypotheses -the false discovery rate, which is equivalent to the FWER when all hypotheses are true but is smaller otherwise.
Book
Model Selection and Multimodel Inference: A Practical Information-Theoretic Approach
TL;DR: The second edition of this book is unique in that it focuses on methods for making formal statistical inference from all the models in an a priori set (Multi-Model Inference).
Book
Multiple Regression: Testing and Interpreting Interactions
Leona S. Aiken,Stephen G. West +1 more
TL;DR: In this article, the effects of predictor scaling on the coefficients of regression equations are investigated. But, they focus mainly on the effect of predictors scaling on coefficients of regressions.
Book
Discovering Statistics Using SPSS
Andy P. Field,Jeremy N.V. Miles +1 more
TL;DR: Suitable for those new to statistics as well as students on intermediate and more advanced courses, the book walks students through from basic to advanced level concepts, all the while reinforcing knowledge through the use of SAS(R).
Journal ArticleDOI
A Simple Sequentially Rejective Multiple Test Procedure
TL;DR: In this paper, a simple and widely accepted multiple test procedure of the sequentially rejective type is presented, i.e. hypotheses are rejected one at a time until no further rejections can be done.