Showing papers by "Fiona Fidler published in 2014"

Yes, but Don’t Underestimate Estimation: Reply to Morey, Rouder, Verhagen, and Wagenmakers (2014)

Abstract: Morey, Rouder, Verhagen, and Wagenmakers (2014) make a useful contribution by emphasizing that hypothesis testing has an important role in science, quite apart from null-hypothesis significance testing, which they agree should not be used. We are encouraged that they describe estimation as essential (p. 1289), and part of what is “becoming the conventional wisdom in psychology” (p. 1289), even if a touch of irony is intended in that wording. However, we do not agree that “the benefits of estimation have been overstated” (p. 1289). Estimation can contribute in many ways, including to model evaluation and theory development. Morey et al. state that “hypothesis testing, not estimation, is necessary for testing the quantitative predictions of theories” (p. 1290). We agree that testing such predictions is necessary, but any approach to assessing a quantitative prediction presumably requires comparison with a relevant quantitative statistic derived from data, which is an estimate by another name. Morey et al. need to explain how what they refer to as hypothesis testing can assess quantitative predictions without some contribution from estimation. If estimation cannot contribute, their concept of hypothesis testing may be narrower than ours. Morey et al. make several bold, prescriptive statements— in addition to this one—about the way science (including hypothesis testing and theory development) should operate. They have little space to expand on their philosophy of science or definition of hypothesis testing, but without more justification, their statements are unconvincing. Cumming (2014, p. 21) referred with approval to a study by Velicer et al. (2008), who used estimation to evaluate the transtheoretical model of behavior change. Velicer et al. judged that 11 of 15 predictions falling within 95% confidence intervals based on data provided overall support for the model. Morey et al. are critical of the arbitrariness of basing the conclusion on 11 of 15 correct predictions and also of the neglect of results expected if the model under test were false. However, judgments about the strengths and weaknesses of a model are inevitably and properly somewhat subjective and therefore somewhat arbitrary. Velicer et al. considered the transtheoretical model without explicit reference to any alternative because no other model with anything like the same promise has been proposed. In our view, they made appropriate use of estimation to evaluate the model and guide its further development. Such knowledgeable judgment in context is a necessary part of any evaluation of a model against data. For example, Bayesian model comparison, which Morey et al. advocate (p. 1290), uses the Bayes factor to quantify the relative strength of evidence that a data set provides for two competing models. Researchers need to judge how Bayes factor values are used to guide conclusions— with or without reference to suggested benchmarks for which values of the Bayes factor provide “barely worth mentioning,” “substantial,” or “strong” evidence for one model over another ( Jeffreys, 1961, p. 432). In fact, an often-cited advantage of Bayesian methods is that they make salient the subjectivity inherent in scientific inference (Berger & Berry, 1988; Wagenmakers, Lee, Lodewyckx, & Iverson, 2008). Similarly, the informationtheoretic approach of Burnham and Anderson (2002), mentioned by Morey et al. (p. 1290), provides Akaike information criterion (AIC) values that allow a quantitative comparison of two or more models, given a set of data. In this case, also, researchers need to take knowledgeable account of the full research context when they decide what conclusions are justified by particular AIC values. 532658 PSSXXX10.1177/0956797614532658Fidler, CummingReply to Morey, Rouder, Verhagen, and Wagenmakers (2014) research-article2014