Many psychologists do not realize that exploratory use of the popular multiway analysis of variance harbors a multiple-comparison problem. In the case of two factors, three separate null hypotheses are subject to test (i.e., two main effects and one interaction). Consequently, the probability of at least one Type I error (if all null hypotheses are true) is 14 % rather than 5 %, if the three tests are independent. We explain the multiple-comparison problem and demonstrate that researchers almost never correct for it. To mitigate the problem, we describe four remedies: the omnibus F test, control of the familywise error rate, control of the false discovery rate, and preregistration of the hypotheses.

/pdf/hidden-multiplicity-in-exploratory-multiway-anova-prevalence-5232rl2agt.pdf

Hidden multiplicity in exploratory multiway ANOVA: Prevalence and remedies.

Learning about hypothesis evaluation using the Bayes factor could enhance psychological research. In contrast to null-hypothesis significance testing it renders the evidence in favor of each of the hypotheses under consideration (it can be used to quantify support for the null-hypothesis) instead of a dichotomous reject/do-not-reject decision; it can straightforwardly be used for the evaluation of multiple hypotheses without having to bother about the proper manner to account for multiple testing; and it allows continuous reevaluation of hypotheses after additional data have been collected (Bayesian updating). This tutorial addresses researchers considering to evaluate their hypotheses by means of the Bayes factor. The focus is completely applied and each topic discussed is illustrated using Bayes factors for the evaluation of hypotheses in the context of an ANOVA model, obtained using the R package bain. Readers can execute all the analyses presented while reading this tutorial if they download bain and the R-codes used. It will be elaborated in a completely nontechnical manner: what the Bayes factor is, how it can be obtained, how Bayes factors should be interpreted, and what can be done with Bayes factors. After reading this tutorial and executing the associated code, researchers will be able to use their own data for the evaluation of hypotheses by means of the Bayes factor, not only in the context of ANOVA models, but also in the context of other statistical models. (PsycINFO Database Record (c) 2019 APA, all rights reserved).

/pdf/a-tutorial-on-testing-hypotheses-using-the-bayes-factor-4tng593674.pdf

A tutorial on testing hypotheses using the Bayes factor.

Many empirical researchers do not realize that the common multiway analysis of variance (ANOVA) harbors a multiple comparison problem. In the case of two factors, three separate null hypotheses are subject to test (i.e., two main effects and one interaction). Consequently, the probability of at least one Type I error (if all null hypotheses are true) is 14% rather than 5% if the three tests are independent. We explain the multiple comparison problem and demonstrate that researchers almost never correct for it. We describe one of several correction procedures (i.e., sequential Bonferroni), and show that its application alters at least one of the substantive conclusions in 45 out of 60 articles considered. An additional method to mitigate the multiplicity in multiway ANOVA is preregistration of hypotheses.

Hidden Multiplicity in Multiway ANOVA: Prevalence, Consequences, and Remedies

Many psychologists do not realize that exploratory use of the popular multiway analysis of variance (ANOVA) harbors a multiple comparison problem. In the case of two factors, three separate null hypotheses are subject to test (i.e., two main effects and one interaction). Consequently, the probability of at least one Type I error (if all null hypotheses are true) is 14% rather than 5% if the three tests are independent. We explain the multiple comparison problem and demonstrate that researchers almost never correct for it. To mitigate the problem, we describe four remedies: the omnibus F test, the control of familywise error rate, the control of false discovery rate, and the preregistration of hypotheses.

Hidden Multiplicity in Multiway ANOVA: Prevalence and Remedies

Exerting effort in a first task can impair self-control performance in a subsequent task. Hundreds of studies have examined this ego depletion effect, but the underlying mechanisms are still unknown. By contrasting the two most prominent models, the strength model and the process model, the following question takes centre stage: Do participants fail to exert self-control because they run short of an unspecified resource or because they lack the motivation to engage in the subsequent task? We contrasted competing predictions (N = 560) from these two models by manipulating monetary incentives to be donated to charity in the first of two tasks. We found evidence of the standard ego depletion effect-self-control performance was impaired after a high- versus a low-demand task in the no-incentive conditions. Incentives had an unexpected effect: Whereas participants in the incentive conditions showed higher intrinsic, autonomous motivation, they did not exert greater effort. This unexpected finding limited the applicability of our registered predictions; thus, we opted to test updated predictions. We discuss the theoretical implications of our understanding of the processes underlying ego depletion effects and their meaning for the ongoing debate about replicability and robustness.

https://onlinelibrary.wiley.com/doi/pdfdirect/10.1111/bjso.12594

Do we fail to exert self-control because we lack resources or motivation? Competing theories to explain a debated phenomenon.

Experimental studies are usually designed with specific expectations about the results in mind. However, most researchers apply some form of omnibus test to test for any differences, with follow up tests like pairwise comparisons or simple effects analyses for further investigation of the effects. The power to find full support for the theory with such an exploratory approach which is usually based on multiple testing is, however, rather disappointing. With the simulations in this paper we showed that many of the common choices in hypothesis testing led to a severely underpowered form of theory evaluation. Furthermore, some less commonly used approaches were presented and a comparison of results in terms of power to find support for the theory was made. We concluded that confirmatory methods are required in the context of theory evaluation and that the scientific literature would benefit from a clearer distinction between confirmatory and exploratory findings. Also, we emphasis the importance of reporting all tests, significant or not, including the appropriate sample statistics like means and standard deviations. Another recommendation is related to the fact that researchers, when they discuss the conclusions of their own study, seem to underestimate the role of sampling variability. The execution of more replication studies in combination with proper reporting of all results provides insight in between study variability and the amount of chance findings.

/pdf/confirmatory-methods-or-huge-samples-are-required-to-obtain-3iadu8lcqd.pdf

Laura Post

Papers

Confirmatory Methods, or Huge Samples, Are Required to Obtain Power for the Evaluation of Theories