Cryptic multiple hypotheses testing in linear models: overestimated effect sizes and the winner's curse.
Reads0
Chats0
TLDR
Full model tests and P value adjustments can be used as a guide to how frequently type I errors arise by sampling variation alone, and favour the presentation of full models, since they best reflect the range of predictors investigated and ensure a balanced representation also of non-significant results.Abstract:
Fitting generalised linear models (GLMs) with more than one predictor has become the standard method of analysis in evolutionary and behavioural research. Often, GLMs are used for exploratory data analysis, where one starts with a complex full model including interaction terms and then simplifies by removing non-significant terms. While this approach can be useful, it is problematic if significant effects are interpreted as if they arose from a single a priori hypothesis test. This is because model selection involves cryptic multiple hypothesis testing, a fact that has only rarely been acknowledged or quantified. We show that the probability of finding at least one ‘significant’ effect is high, even if all null hypotheses are true (e.g. 40% when starting with four predictors and their two-way interactions). This probability is close to theoretical expectations when the sample size (N) is large relative to the number of predictors including interactions (k). In contrast, type I error rates strongly exceed even those expectations when model simplification is applied to models that are over-fitted before simplification (low N/k ratio). The increase in false-positive results arises primarily from an overestimation of effect sizes among significant predictors, leading to upward-biased effect sizes that often cannot be reproduced in follow-up studies (‘the winner's curse’). Despite having their own problems, full model tests and P value adjustments can be used as a guide to how frequently type I errors arise by sampling variation alone. We favour the presentation of full models, since they best reflect the range of predictors investigated and ensure a balanced representation also of non-significant results.read more
Citations
More filters
Journal ArticleDOI
Who is who matters—The effects of pseudoreplication in stable isotope analyses
Roger Mundry,Viktoria M. Oelze +1 more
TL;DR: It is shown that pseudoreplication can severely affect the probability of erroneous significance as well as non‐significance in stable isotope data of great ape hair, and several strategies to avoid pseudorePLication in primate isotope ecology are suggested.
Journal ArticleDOI
Endocrine changes related to dog domestication: Comparing urinary cortisol and oxytocin in hand-raised, pack-living dogs and wolves
Gwendolyn Wirobski,Friederike Range,F. S. Schaebs,Rupert Palme,Tobias Deschner,Sarah Marshall-Pescini +5 more
TL;DR: Analysis of urine samples of hand-raised, pack-living domestic dogs and their non-domestic relatives, grey wolves found higher cortisol concentrations in dogs than wolves, and oxytocin concentrations were higher in dogs compared to wolves although the effect was relatively small.
Journal ArticleDOI
Direct benefits from choosing a virgin male in the European grapevine moth, Lobesia botrana
TL;DR: The results suggest that females are able to discriminate between males with different mating experience, and prefer virgin males, thereby maximizing direct benefits associated with receiving large spermatophores.
Journal ArticleDOI
Acute psychosocial stress alters thalamic network centrality
Janis Reinelt,Marie Uhlig,Karsten Müller,Mark E. Lauckner,Deniz Kumral,H. Lina Schaare,Blazej M. Baczkowski,Anahit Babayan,Miray Erbey,Josefin Roebbig,Andrea M. F. Reiter,Yoon-Ju Bae,Juergen Kratzsch,Joachim Thiery,Talma Hendler,Arno Villringer,Michael Gaebler +16 more
TL;DR: It is concluded that thalamic areas are central for information processing after stress exposure and may provide an interface for the stress response in the rest of the body and in the mind.
Journal ArticleDOI
Testosterone production, sexually dimorphic morphology, and digit ratio in the dark-eyed junco
TL;DR: It is concluded that individual variation in exposure to developmental hormones, as reflected by 2D:4D, is correlated with adult hormone production ability and sexually dimorphic morphology in adulthood, suggesting that endogenous variation in steroid hormone exposure may have long-term consequences similar to those seen in experimental manipulations.
References
More filters
Journal ArticleDOI
Controlling the false discovery rate: a practical and powerful approach to multiple testing
Yoav Benjamini,Yosef Hochberg +1 more
TL;DR: In this paper, a different approach to problems of multiple significance testing is presented, which calls for controlling the expected proportion of falsely rejected hypotheses -the false discovery rate, which is equivalent to the FWER when all hypotheses are true but is smaller otherwise.
Book
Model Selection and Multimodel Inference: A Practical Information-Theoretic Approach
TL;DR: The second edition of this book is unique in that it focuses on methods for making formal statistical inference from all the models in an a priori set (Multi-Model Inference).
Book
Multiple Regression: Testing and Interpreting Interactions
Leona S. Aiken,Stephen G. West +1 more
TL;DR: In this article, the effects of predictor scaling on the coefficients of regression equations are investigated. But, they focus mainly on the effect of predictors scaling on coefficients of regressions.
Book
Discovering Statistics Using SPSS
Andy P. Field,Jeremy N.V. Miles +1 more
TL;DR: Suitable for those new to statistics as well as students on intermediate and more advanced courses, the book walks students through from basic to advanced level concepts, all the while reinforcing knowledge through the use of SAS(R).
Journal ArticleDOI
A Simple Sequentially Rejective Multiple Test Procedure
TL;DR: In this paper, a simple and widely accepted multiple test procedure of the sequentially rejective type is presented, i.e. hypotheses are rejected one at a time until no further rejections can be done.