Cryptic multiple hypotheses testing in linear models: overestimated effect sizes and the winner's curse.
Reads0
Chats0
TLDR
Full model tests and P value adjustments can be used as a guide to how frequently type I errors arise by sampling variation alone, and favour the presentation of full models, since they best reflect the range of predictors investigated and ensure a balanced representation also of non-significant results.Abstract:
Fitting generalised linear models (GLMs) with more than one predictor has become the standard method of analysis in evolutionary and behavioural research. Often, GLMs are used for exploratory data analysis, where one starts with a complex full model including interaction terms and then simplifies by removing non-significant terms. While this approach can be useful, it is problematic if significant effects are interpreted as if they arose from a single a priori hypothesis test. This is because model selection involves cryptic multiple hypothesis testing, a fact that has only rarely been acknowledged or quantified. We show that the probability of finding at least one ‘significant’ effect is high, even if all null hypotheses are true (e.g. 40% when starting with four predictors and their two-way interactions). This probability is close to theoretical expectations when the sample size (N) is large relative to the number of predictors including interactions (k). In contrast, type I error rates strongly exceed even those expectations when model simplification is applied to models that are over-fitted before simplification (low N/k ratio). The increase in false-positive results arises primarily from an overestimation of effect sizes among significant predictors, leading to upward-biased effect sizes that often cannot be reproduced in follow-up studies (‘the winner's curse’). Despite having their own problems, full model tests and P value adjustments can be used as a guide to how frequently type I errors arise by sampling variation alone. We favour the presentation of full models, since they best reflect the range of predictors investigated and ensure a balanced representation also of non-significant results.read more
Citations
More filters
Journal ArticleDOI
Female chimpanzee associations with male kin: trade-offs between inbreeding avoidance and infanticide protection
TL;DR: In this paper , the authors used 43 years of behavioural data on the wild chimpanzees, Pan troglodytes schweinfurthii, of Gombe National Park, Tanzania, to examine association patterns between females and their adult maternal male kin.
Journal ArticleDOI
Appeasement function of displacement behaviours? Dogs’ behavioural displays exhibited towards threatening and neutral humans
TL;DR: In this paper , the attitude of the dogs towards the strangers was categorized as reactive and non-reactive, i.e., barking and lunging towards the stimulus, and remaining passive in front of the stimuli.
Journal ArticleDOI
An Investigation into the Influence of Different Types of Nesting Materials upon the Welfare of Captive Chimpanzees (Pan troglodytes)
TL;DR: The authors investigated the impact of different nesting materials (leaves and branches, long grass, cotton sheets, and shredded newspaper) upon the welfare of chimpanzees at Tacugama Chimpanzee Sanctuary (Sierra Leone).
Journal ArticleDOI
Dogs fail to reciprocate the receipt of food from a human in a food-giving task.
Jim McGetrick,Lisa Poncet,Lisa Poncet,Lisa Poncet,Marietta Amann,Johannes Schullern-Schrattenhofen,Leona Fux,Mayte Martínez,Friederike Range +8 more
TL;DR: In this paper, the authors investigated whether dogs reciprocate the receipt of food from humans and found that dogs do not recognize the cooperative or uncooperative act of the humans during the experience phase.
Journal ArticleDOI
Sex Differences in the Swordtail Xiphophorus hellerii Revealed by a New Method to Investigate Volumetric Data.
TL;DR: A new method is introduced to compare the relative volumes of parts that does not need a scaling reference and transforms the absolute part volumes into a ratio matrix (volume ratio transformation, VRT), which is free of any scaling factors and can be used to compare groups of animals.
References
More filters
Journal ArticleDOI
Controlling the false discovery rate: a practical and powerful approach to multiple testing
Yoav Benjamini,Yosef Hochberg +1 more
TL;DR: In this paper, a different approach to problems of multiple significance testing is presented, which calls for controlling the expected proportion of falsely rejected hypotheses -the false discovery rate, which is equivalent to the FWER when all hypotheses are true but is smaller otherwise.
Book
Model Selection and Multimodel Inference: A Practical Information-Theoretic Approach
TL;DR: The second edition of this book is unique in that it focuses on methods for making formal statistical inference from all the models in an a priori set (Multi-Model Inference).
Book
Multiple Regression: Testing and Interpreting Interactions
Leona S. Aiken,Stephen G. West +1 more
TL;DR: In this article, the effects of predictor scaling on the coefficients of regression equations are investigated. But, they focus mainly on the effect of predictors scaling on coefficients of regressions.
Book
Discovering Statistics Using SPSS
Andy P. Field,Jeremy N.V. Miles +1 more
TL;DR: Suitable for those new to statistics as well as students on intermediate and more advanced courses, the book walks students through from basic to advanced level concepts, all the while reinforcing knowledge through the use of SAS(R).
Journal ArticleDOI
A Simple Sequentially Rejective Multiple Test Procedure
TL;DR: In this paper, a simple and widely accepted multiple test procedure of the sequentially rejective type is presented, i.e. hypotheses are rejected one at a time until no further rejections can be done.