False-Positive Psychology: Undisclosed Flexibility in Data Collection and Analysis Allows Presenting Anything as Significant
Reads0
Chats0
TLDR
It is shown that despite empirical psychologists’ nominal endorsement of a low rate of false-positive findings, flexibility in data collection, analysis, and reporting dramatically increases actual false- positive rates, and a simple, low-cost, and straightforwardly effective disclosure-based solution is suggested.Abstract:
In this article, we accomplish two things. First, we show that despite empirical psychologists' nominal endorsement of a low rate of false-positive findings (≤ .05), flexibility in data collection, analysis, and reporting dramatically increases actual false-positive rates. In many cases, a researcher is more likely to falsely find evidence that an effect exists than to correctly find evidence that it does not. We present computer simulations and a pair of actual experiments that demonstrate how unacceptably easy it is to accumulate (and report) statistically significant evidence for a false hypothesis. Second, we suggest a simple, low-cost, and straightforwardly effective disclosure-based solution to this problem. The solution involves six concrete requirements for authors and four guidelines for reviewers, all of which impose a minimal burden on the publication process.read more
Citations
More filters
Journal ArticleDOI
Shifts in Methodology and Theory in Menstrual Cycle Research on Attraction
Abstract: This paper critically examines the hypothesis that different phases of the menstrual cycle induce changes in women’s mate preferences Empirically, we show that literature on this topic may be particularly prone to experimenter degrees of freedom, in which experimenters increase their likelihood of finding significant effects through elasticity in methodological and analytical strategies (eg, flexibility in calculation of fertile and nonfertile phases, exclusion criteria, moderators, and analysis of dependent variables) Theoretically, we address misconceptions presented by Gildersleeve and colleagues (2013a) We reveal inconsistencies in the theoretical foundation for this work and discuss tension between theory and data In short, there is sound reason to question whether reported menstrual cycle effects in women’s mate preferences are indeed real
Journal ArticleDOI
Comparing the accuracy of experimental estimates to guessing: a new perspective on replication and the "Crisis of Confidence" in psychology.
TL;DR: A general measure of estimation accuracy for fundamental research designs, called v, which compares the estimation accuracy of the ubiquitous ordinary least squares estimator with a benchmark estimator that randomizes the direction of treatment effects.
Journal ArticleDOI
Predicting Early-Childhood Gender Transitions:
James R. Rae,James R. Rae,Selin Gülgöz,Lily Durwood,Madeleine DeMeules,Riley Lowe,Gabrielle Lindquist,Kristina R. Olson +7 more
TL;DR: It is found that stronger cross-sex identification and preferences expressed by gender-nonconforming children at initial testing predicted whether they later socially transitioned, and social transitions may be predictable from gender identification and preference.
Journal ArticleDOI
Mega-analysis methods in ENIGMA: The experience of the generalized anxiety disorder working group.
André Zugman,Anita Harrewijn,Elise M. Cardinale,Hannah Zwiebel,Gabrielle F. Freitag,Katy E. Werwath,Janna Marie Bas-Hoogendam,Janna Marie Bas-Hoogendam,Nynke A. Groenewold,Moji Aghajani,Kevin Hilbert,Narcís Cardoner,Daniel Porta-Casteràs,Savannah N. Gosnell,Ramiro Salas,Karina S. Blair,James R. Blair,Mira Z. Hammoud,Mohammed R. Milad,Katie L. Burkhouse,K. Luan Phan,Heidi K. Schroeder,Jeffrey R. Strawn,Katja Beesdo-Baum,Sophia I. Thomopoulos,Hans J. Grabe,Hans J. Grabe,Sandra Van der Auwera,Sandra Van der Auwera,Katharina Wittfeld,Katharina Wittfeld,Jared A. Nielsen,Randy L. Buckner,Jordan W. Smoller,Benson Mwangi,Jair C. Soares,Mon-Ju Wu,Giovana Zunta-Soares,Andrea Parolin Jackowski,Pedro Mario Pan,Giovanni Abrahão Salum,Michal Assaf,Michal Assaf,Gretchen J. Diefenbach,Gretchen J. Diefenbach,Paolo Brambilla,Eleonora Maggioni,David Hofmann,Thomas Straube,Carmen Andreescu,Rachel Berta,Erica Tamburo,Rebecca B. Price,Gisele Gus Manfro,Hugo D. Critchley,Elena Makovac,Matteo Mancini,Frances Meeten,Cristina Ottaviani,Federica Agosta,Elisa Canu,Camilla Cividini,Massimo Filippi,Milutin Kostić,Ana Munjiza,Courtney A. Filippi,Ellen Leibenluft,Bianca A.V. Alberton,Nicholas L. Balderston,Monique Ernst,Christian Grillon,Lilianne R. Mujica-Parodi,Helena van Nieuwenhuizen,Gregory A. Fonzo,Martin P. Paulus,Murray B. Stein,Raquel E. Gur,Ruben C. Gur,Antonia N. Kaczkurkin,Bart Larsen,Theodore D. Satterthwaite,Jennifer Harper,Michael J. Myers,Michael T. Perino,Qiongru Yu,Chad M. Sylvester,Dick J. Veltman,Ulrike Lueken,Nic J.A. van der Wee,Nic J.A. van der Wee,Dan J. Stein,Neda Jahanshad,Paul M. Thompson,Daniel S. Pine,Anderson M. Winkler +94 more
TL;DR: The background information and rationale for the various methodological decisions, as well as the approach taken to implement them are summarized to document the approach and help guide other research groups working with large brain imaging data sets as they develop their own analytic pipelines for mega‐analyses.
Journal ArticleDOI
The psychology of experimental psychologists: Overcoming cognitive constraints to improve research: The 47th Sir Frederic Bartlett Lecture.
TL;DR: It is argued that if proposed solutions to the replication crisis are to be effective, they need to take into account human cognitive constraints that can distort all stages of the research process, including design and execution of experiments, analysis of data, and writing up findings for publication.
References
More filters
Journal ArticleDOI
The case for motivated reasoning.
TL;DR: It is proposed that motivation may affect reasoning through reliance on a biased set of cognitive processes--that is, strategies for accessing, constructing, and evaluating beliefs--that are considered most likely to yield the desired conclusion.
Why Most Published Research Findings Are False
TL;DR: In this paper, the authors discuss the implications of these problems for the conduct and interpretation of research and suggest that claimed research findings may often be simply accurate measures of the prevailing bias.
Journal ArticleDOI
Group sequential methods in the design and analysis of clinical trials
TL;DR: In this article, a group sequential design is proposed to divide patient entry into a number of equal-sized groups so that the decision to stop the trial or continue is based on repeated significance tests of the accumulated data after each group is evaluated.
Journal ArticleDOI
Measuring the Prevalence of Questionable Research Practices With Incentives for Truth Telling
TL;DR: It is found that the percentage of respondents who have engaged in questionable practices was surprisingly high, which suggests that some questionable practices may constitute the prevailing research norm.
Journal ArticleDOI
Attribution of success and failure revisited, or: The motivational bias is alive and well in attribution theory
TL;DR: The authors found that self-serving effects for both success and failure are obtained in most but not all experimental paradigms, and that these attributions are better understood in motivational than in information-processing terms.
Related Papers (5)
Estimating the reproducibility of psychological science
Alexander A. Aarts,Joanna E. Anderson,Christopher J. Anderson,Peter Raymond Attridge,Peter Raymond Attridge,Angela S. Attwood,Jordan Axt,Molly Babel,Štěpán Bahník,Erica Baranski,Michael Barnett-Cowan,Elizabeth Bartmess,Jennifer S. Beer,Raoul Bell,Heather Bentley,Leah Beyan,Grace Binion,Grace Binion,Denny Borsboom,Annick Bosch,Frank A. Bosco,Sara Bowman,Mark J. Brandt,Erin L Braswell,Hilmar Brohmer,Benjamin T. Brown,Kristina G. Brown,Jovita Brüning,Jovita Brüning,Ann Calhoun-Sauls,Shannon P. Callahan,Elizabeth Chagnon,Jesse Chandler,Jesse Chandler,Christopher R. Chartier,Felix Cheung,Felix Cheung,Cody D. Christopherson,Linda Cillessen,Russ Clay,Hayley M. D. Cleary,Mark D. Cloud,Michael Conn,Johanna Cohoon,Simon Columbus,Andreas Cordes,Giulio Costantini,Leslie Cramblet Alvarez,Ed Cremata,Jan Crusius,Jamie DeCoster,Michelle A. DeGaetano,Nicolás Delia Penna,Bobby Den Bezemer,Marie K. Deserno,Olivia Devitt,Laura Dewitte,David G. Dobolyi,Geneva T. Dodson,M. Brent Donnellan,Ryan Donohue,Rebecca A. Dore,Angela Rachael Dorrough,Angela Rachael Dorrough,Anna Dreber,Michelle Dugas,Elizabeth W. Dunn,Kayleigh E Easey,Sylvia Eboigbe,Casey Eggleston,Jo Embley,Sacha Epskamp,Timothy M. Errington,Vivien Estel,Frank J. Farach,Jenelle Feather,Anna Fedor,Belén Fernández-Castilla,Susann Fiedler,James G. Field,Stanka A. Fitneva,Taru Flagan,Amanda L. Forest,Eskil Forsell,Joshua D. Foster,Michael C. Frank,Rebecca S. Frazier,Heather M. Fuchs,Philip A. Gable,Jeff Galak,Elisa Maria Galliani,Anup Gampa,Sara García,Douglas Gazarian,Elizabeth Gilbert,Roger Giner-Sorolla,Andreas Glöckner,Andreas Glöckner,Lars Goellner,Jin X. Goh,Rebecca M. Goldberg,Patrick T. Goodbourn,Shauna Gordon-McKeon,Bryan Gorges,Jessie Gorges,Justin Goss,Jesse Graham,James A. Grange,Jeremy R. Gray,Chris H.J. Hartgerink,Joshua K. Hartshorne,Fred Hasselman,Timothy Hayes,Emma Heikensten,Felix Henninger,Felix Henninger,John Hodsoll,Taylor Holubar,Gea Hoogendoorn,Denise J. Humphries,Cathy On-Ying Hung,Nathali Immelman,Vanessa C. Irsik,Georg Jahn,Frank Jäkel,Marc Jekel,Magnus Johannesson,Larissa Gabrielle Johnson,David J. Johnson,Kate M. Johnson,William J. Johnston,Kai J. Jonas,Jennifer A. Joy-Gaba,Heather Barry Kappes,Kim Kelso,Mallory C. Kidwell,Seung K. Kim,Matthew W. Kirkhart,Bennett Kleinberg,Bennett Kleinberg,Goran Knežević,Franziska Maria Kolorz,Jolanda J. Kossakowski,Robert Krause,Job Krijnen,Tim Kuhlmann,Yoram K. Kunkels,Megan M. Kyc,Calvin K. Lai,Aamir Laique,Daniel Lakens,Kristin A. Lane,Bethany Lassetter,Ljiljana B. Lazarević,Etienne P. Le Bel,Key Jung Lee,Minha Lee,Kristi M. Lemm,Carmel A. Levitan,Melissa Lewis,Lin Lin,Stephanie C. Lin,Matthias Lippold,Darren Loureiro,Ilse Luteijn,Sean P. Mackinnon,Heather N. Mainard,Denise C. Marigold,Daniel P. Martin,Tylar Martinez,E. J. Masicampo,Joshua J. Matacotta,Maya B. Mathur,Michael May,Michael May,Nicole Mechin,Pranjal H. Mehta,Johannes M. Meixner,Johannes M. Meixner,Alissa Melinger,Jeremy K. Miller,Mallorie Miller,Katherine Moore,Katherine Moore,Marcus Möschl,Matt Motyl,Stephanie M. Müller,Marcus R. Munafò,Koen Ilja Neijenhuijs,Taylor Nervi,Gandalf Nicolas,Gustav Nilsonne,Gustav Nilsonne,Brian A. Nosek,Brian A. Nosek,Michèle B. Nuijten,Catherine Olsson,Catherine Olsson,Colleen Osborne,Lutz Ostkamp,Misha Pavel,Ian S. Penton-Voak,Olivia Perna,Cyril Pernet,Marco Perugini,R. Nathan Pipitone,Michael C. Pitts,Franziska Plessow,Franziska Plessow,Jason M. Prenoveau,Rima-Maria Rahal,Rima-Maria Rahal,Kate A. Ratliff,David Reinhard,Frank Renkewitz,Ashley A. Ricker,Anastasia E. Rigney,Andrew M Rivers,Mark A. Roebke,Abraham M. Rutchick,Robert S. Ryan,Onur Sahin,Anondah R. Saide,Gillian M. Sandstrom,David Santos,David Santos,Rebecca Saxe,René Schlegelmilch,René Schlegelmilch,Kathleen Schmidt,Sabine Scholz,Larissa Seibel,Dylan Selterman,Samuel Shaki,William B. Simpson,H. Colleen Sinclair,Jeanine L. M. Skorinko,Agnieszka Slowik,Joel S. Snyder,Courtney K. Soderberg,Carina Sonnleitner,Nick Spencer,Jeffrey R. Spies,Sara Steegen,Stefan Stieger,Nina Strohminger,Gavin Brent Sullivan,Thomas Talhelm,Megan Tapia,Anniek M. te Dorsthorst,Manuela Thomae,Manuela Thomae,Sarah L. Thomas,Pia Tio,Frits Traets,Steve N.H. Tsang,Francis Tuerlinckx,Paul J. Turchan,Milan Valášek,Anna E. Van't Veer,Robbie C. M. van Aert,Marcel A.L.M. van Assen,Riet van Bork,Mathijs Van De Ven,Don van den Bergh,Marije van der Hulst,Roel van Dooren,Johnny van Doorn,Daan R. van Renswoude,Hedderik van Rijn,Wolf Vanpaemel,Alejandro Vásquez Echeverría,Melissa Vazquez,Natalia Vélez,Marieke Vermue,Mark Verschoor,Michelangelo Vianello,Martin Voracek,Gina Vuu,Eric-Jan Wagenmakers,Joanneke Weerdmeester,Ashlee Welsh,Erin C. Westgate,Joeri Wissink,Michael J. Wood,Andy T. Woods,Andy T. Woods,Emily M. Wright,Sining Wu,Marcel Zeelenberg,Kellylynn Zuni +290 more