False-Positive Psychology: Undisclosed Flexibility in Data Collection and Analysis Allows Presenting Anything as Significant
Reads0
Chats0
TLDR
It is shown that despite empirical psychologists’ nominal endorsement of a low rate of false-positive findings, flexibility in data collection, analysis, and reporting dramatically increases actual false- positive rates, and a simple, low-cost, and straightforwardly effective disclosure-based solution is suggested.Abstract:
In this article, we accomplish two things. First, we show that despite empirical psychologists' nominal endorsement of a low rate of false-positive findings (≤ .05), flexibility in data collection, analysis, and reporting dramatically increases actual false-positive rates. In many cases, a researcher is more likely to falsely find evidence that an effect exists than to correctly find evidence that it does not. We present computer simulations and a pair of actual experiments that demonstrate how unacceptably easy it is to accumulate (and report) statistically significant evidence for a false hypothesis. Second, we suggest a simple, low-cost, and straightforwardly effective disclosure-based solution to this problem. The solution involves six concrete requirements for authors and four guidelines for reviewers, all of which impose a minimal burden on the publication process.read more
Citations
More filters
Journal ArticleDOI
Don't fear 'fear conditioning': Methodological considerations for the design and analysis of studies on human fear acquisition, extinction, and return of fear
Tina B. Lonsdorf,Mareike M. Menz,Marta Andreatta,Miguel A. Fullana,Armita Golkar,Jan Haaker,Ivo Heitland,Andrea Hermann,Manuel Kuhn,Onno Kruse,Shira Meir Drexler,Ann Meulders,Frauke Nees,Andre Pittig,Jan Richter,Sonja Römer,Youssef Shiban,Anja Schmitz,Benjamin Straube,Bram Vervliet,Julia Wendt,Johanna M.P. Baas,Christian J. Merz +22 more
TL;DR: This compendium is intended to provide a basis for the development of a common procedural and terminology framework for the field of human fear conditioning and provides evidence‐based guidance for methodological decisions on study design, outcome measures, and analyses.
Journal ArticleDOI
Threatening communication: a critical re-analysis and a revised meta-analytic test of fear appeal theory
TL;DR: A meta-analysis of six studies that satisfied the inclusion criteria clearly showed a significant interaction between threat and efficacy, such that threat only had an effect under high efficacy, and efficacy only had aeffect under high threat.
Journal ArticleDOI
The Oxytocin Receptor: From Intracellular Signaling to Behavior
Benjamin Jurek,Inga D. Neumann +1 more
TL;DR: The mechanisms of OXT expression and release, expression and binding of the OXTR in brain and periphery, OX TR-coupled signaling cascades, and their involvement in behavioral outcomes are discussed to assemble a comprehensive picture of the central and peripheral OXT system.
Journal Article
How to make more published research true
TL;DR: In this article, a meta-Research Innovation Center at Stanford (METRICS) at Stanford was used to support the creation of a new data collection system for health-related data.
Journal ArticleDOI
Many Labs 2: Investigating Variation in Replicability Across Samples and Settings
Richard A. Klein,Michelangelo Vianello,Fred Hasselman,Byron G. Adams,Reginald B. Adams,Sinan Alper,Mark Aveyard,Jordan Axt,Mayowa T. Babalola,Štěpán Bahník,Rishtee Batra,Mihály Berkics,Michael J. Bernstein,Daniel R. Berry,Olga Bialobrzeska,Evans Dami Binan,Konrad Bocian,Mark J. Brandt,Robert Busching,Anna Cabak Rédei,Huajian Cai,Fanny Cambier,Katarzyna Cantarero,Cheryl L. Carmichael,Francisco Ceric,Jesse Chandler,Jen-Ho Chang,Armand Chatard,Eva E. Chen,Winnee Cheong,David C. Cicero,Sharon Coen,Jennifer A. Coleman,Brian Collisson,Morgan Conway,Katherine S. Corker,Paul G. Curran,Fiery Cushman,Zubairu K. Dagona,Ilker Dalgar,Anna Dalla Rosa,William E. Davis,Maaike J. de Bruijn,Leander De Schutter,Thierry Devos,Marieke de Vries,Marieke de Vries,Canay Doğulu,Nerisa Dozo,Kristin Nicole Dukes,Yarrow Dunham,Kevin Durrheim,Charles R. Ebersole,John E. Edlund,Anja Eller,Alexander S. English,Carolyn Finck,Natalia Frankowska,Miguel-Ángel Freyre,Mike Friedman,Elisa Maria Galliani,Joshua C. Gandi,Tanuka Ghoshal,Steffen R. Giessner,Tripat Gill,Timo Gnambs,Ángel Gómez,Roberto González,Jesse Graham,Jon Grahe,Ivan Grahek,Eva G. T. Green,Kakul Hai,Matthew Haigh,Elizabeth L. Haines,Michael P. Hall,Marie E. Heffernan,Joshua A. Hicks,Petr Houdek,Jeffrey R. Huntsinger,Ho Phi Huynh,Hans IJzerman,Yoel Inbar,Åse Innes-Ker,William Jiménez-Leal,Melissa-Sue John,Jennifer A. Joy-Gaba,Roza G. Kamiloğlu,Heather Barry Kappes,Serdar Karabati,Haruna Karick,Victor N. Keller,Anna Kende,Nicolas Kervyn,Goran Knežević,Carrie Kovacs,Lacy E. Krueger,German Kurapov,Jamie Kurtz,Daniel Lakens,Ljiljana B. Lazarević,Carmel A. Levitan,Neil A. Lewis,Samuel Lincoln Bezerra Lins,Nikolette P. Lipsey,Joy E. Losee,Esther Maassen,Angela T. Maitner,Winfrida Malingumu,Robyn K. Mallett,Satia A. Marotta,Janko Međedović,Fernando Mena-pacheco,Taciano L. Milfont,Wendy L. Morris,Sean C. Murphy,Andriy Myachykov,Nick Neave,Koen Ilja Neijenhuijs,Anthony J. Nelson,Félix Neto,Austin Lee Nichols,Aaron Ocampo,Susan L. O'Donnell,Haruka Oikawa,Masanori Oikawa,Elsie Ong,Gábor Orosz,Małgorzata Osowiecka,Grant Packard,Rolando Pérez-Sánchez,Boban Petrović,Ronaldo Pilati,Brad Pinter,Lysandra Podesta,Gabrielle Pogge,Monique Pollmann,Abraham M. Rutchick,Patricio Saavedra,Alexander K. Saeri,Erika Salomon,Kathleen Schmidt,Felix D. Schönbrodt,Maciej Sekerdej,David Sirlopú,Jeanine L. M. Skorinko,Michael A. Smith,Vanessa Smith-Castro,Karin C.H.J. Smolders,Agata Sobkow,Walter Sowden,Philipp Spachtholz,Manini Srivastava,Troy G. Steiner,Jeroen Stouten,Chris N. H. Street,Oskar K. Sundfelt,Stephanie Szeto,Ewa Szumowska,Andrew C. W. Tang,Norbert K. Tanzer,Morgan J. Tear,Jordan Theriault,Manuela Thomae,David Torres,Jakub Traczyk,Joshua M. Tybur,Adrienn Ujhelyi,Robbie C. M. van Aert,Marcel A.L.M. van Assen,Marije van der Hulst,Paul A. M. Van Lange,Anna van 't Veer,Alejandro Vásquez Echeverría,Leigh Ann Vaughn,Alexandra Vázquez,Luis Diego Vega,Catherine Verniers,Mark Verschoor,Ingrid P. J. Voermans,Marek A. Vranka,Cheryl Alyssa Welch,Aaron L. Wichman,Lisa A. Williams,Michael Wood,Julie A. Woodzicka,Marta Wrońska,Liane Young,John M. Zelenski,Zeng Zhijia,Brian A. Nosek +190 more
TL;DR: This paper conducted preregistered replications of 28 classic and contemporary published findings, with protocols that were peer reviewed in advance, to examine variation in effect magnitudes across samples and settings, and found that very little heterogeneity was attributable to the order in which the tasks were performed or whether the task were administered in lab versus online.
References
More filters
Journal ArticleDOI
The case for motivated reasoning.
TL;DR: It is proposed that motivation may affect reasoning through reliance on a biased set of cognitive processes--that is, strategies for accessing, constructing, and evaluating beliefs--that are considered most likely to yield the desired conclusion.
Why Most Published Research Findings Are False
TL;DR: In this paper, the authors discuss the implications of these problems for the conduct and interpretation of research and suggest that claimed research findings may often be simply accurate measures of the prevailing bias.
Journal ArticleDOI
Group sequential methods in the design and analysis of clinical trials
TL;DR: In this article, a group sequential design is proposed to divide patient entry into a number of equal-sized groups so that the decision to stop the trial or continue is based on repeated significance tests of the accumulated data after each group is evaluated.
Journal ArticleDOI
Measuring the Prevalence of Questionable Research Practices With Incentives for Truth Telling
TL;DR: It is found that the percentage of respondents who have engaged in questionable practices was surprisingly high, which suggests that some questionable practices may constitute the prevailing research norm.
Journal ArticleDOI
Attribution of success and failure revisited, or: The motivational bias is alive and well in attribution theory
TL;DR: The authors found that self-serving effects for both success and failure are obtained in most but not all experimental paradigms, and that these attributions are better understood in motivational than in information-processing terms.
Related Papers (5)
Estimating the reproducibility of psychological science
Alexander A. Aarts,Joanna E. Anderson,Christopher J. Anderson,Peter Raymond Attridge,Peter Raymond Attridge,Angela S. Attwood,Jordan Axt,Molly Babel,Štěpán Bahník,Erica Baranski,Michael Barnett-Cowan,Elizabeth Bartmess,Jennifer S. Beer,Raoul Bell,Heather Bentley,Leah Beyan,Grace Binion,Grace Binion,Denny Borsboom,Annick Bosch,Frank A. Bosco,Sara Bowman,Mark J. Brandt,Erin L Braswell,Hilmar Brohmer,Benjamin T. Brown,Kristina G. Brown,Jovita Brüning,Jovita Brüning,Ann Calhoun-Sauls,Shannon P. Callahan,Elizabeth Chagnon,Jesse Chandler,Jesse Chandler,Christopher R. Chartier,Felix Cheung,Felix Cheung,Cody D. Christopherson,Linda Cillessen,Russ Clay,Hayley M. D. Cleary,Mark D. Cloud,Michael Conn,Johanna Cohoon,Simon Columbus,Andreas Cordes,Giulio Costantini,Leslie Cramblet Alvarez,Ed Cremata,Jan Crusius,Jamie DeCoster,Michelle A. DeGaetano,Nicolás Delia Penna,Bobby Den Bezemer,Marie K. Deserno,Olivia Devitt,Laura Dewitte,David G. Dobolyi,Geneva T. Dodson,M. Brent Donnellan,Ryan Donohue,Rebecca A. Dore,Angela Rachael Dorrough,Angela Rachael Dorrough,Anna Dreber,Michelle Dugas,Elizabeth W. Dunn,Kayleigh E Easey,Sylvia Eboigbe,Casey Eggleston,Jo Embley,Sacha Epskamp,Timothy M. Errington,Vivien Estel,Frank J. Farach,Jenelle Feather,Anna Fedor,Belén Fernández-Castilla,Susann Fiedler,James G. Field,Stanka A. Fitneva,Taru Flagan,Amanda L. Forest,Eskil Forsell,Joshua D. Foster,Michael C. Frank,Rebecca S. Frazier,Heather M. Fuchs,Philip A. Gable,Jeff Galak,Elisa Maria Galliani,Anup Gampa,Sara García,Douglas Gazarian,Elizabeth Gilbert,Roger Giner-Sorolla,Andreas Glöckner,Andreas Glöckner,Lars Goellner,Jin X. Goh,Rebecca M. Goldberg,Patrick T. Goodbourn,Shauna Gordon-McKeon,Bryan Gorges,Jessie Gorges,Justin Goss,Jesse Graham,James A. Grange,Jeremy R. Gray,Chris H.J. Hartgerink,Joshua K. Hartshorne,Fred Hasselman,Timothy Hayes,Emma Heikensten,Felix Henninger,Felix Henninger,John Hodsoll,Taylor Holubar,Gea Hoogendoorn,Denise J. Humphries,Cathy On-Ying Hung,Nathali Immelman,Vanessa C. Irsik,Georg Jahn,Frank Jäkel,Marc Jekel,Magnus Johannesson,Larissa Gabrielle Johnson,David J. Johnson,Kate M. Johnson,William J. Johnston,Kai J. Jonas,Jennifer A. Joy-Gaba,Heather Barry Kappes,Kim Kelso,Mallory C. Kidwell,Seung K. Kim,Matthew W. Kirkhart,Bennett Kleinberg,Bennett Kleinberg,Goran Knežević,Franziska Maria Kolorz,Jolanda J. Kossakowski,Robert Krause,Job Krijnen,Tim Kuhlmann,Yoram K. Kunkels,Megan M. Kyc,Calvin K. Lai,Aamir Laique,Daniel Lakens,Kristin A. Lane,Bethany Lassetter,Ljiljana B. Lazarević,Etienne P. Le Bel,Key Jung Lee,Minha Lee,Kristi M. Lemm,Carmel A. Levitan,Melissa Lewis,Lin Lin,Stephanie C. Lin,Matthias Lippold,Darren Loureiro,Ilse Luteijn,Sean P. Mackinnon,Heather N. Mainard,Denise C. Marigold,Daniel P. Martin,Tylar Martinez,E. J. Masicampo,Joshua J. Matacotta,Maya B. Mathur,Michael May,Michael May,Nicole Mechin,Pranjal H. Mehta,Johannes M. Meixner,Johannes M. Meixner,Alissa Melinger,Jeremy K. Miller,Mallorie Miller,Katherine Moore,Katherine Moore,Marcus Möschl,Matt Motyl,Stephanie M. Müller,Marcus R. Munafò,Koen Ilja Neijenhuijs,Taylor Nervi,Gandalf Nicolas,Gustav Nilsonne,Gustav Nilsonne,Brian A. Nosek,Brian A. Nosek,Michèle B. Nuijten,Catherine Olsson,Catherine Olsson,Colleen Osborne,Lutz Ostkamp,Misha Pavel,Ian S. Penton-Voak,Olivia Perna,Cyril Pernet,Marco Perugini,R. Nathan Pipitone,Michael C. Pitts,Franziska Plessow,Franziska Plessow,Jason M. Prenoveau,Rima-Maria Rahal,Rima-Maria Rahal,Kate A. Ratliff,David Reinhard,Frank Renkewitz,Ashley A. Ricker,Anastasia E. Rigney,Andrew M Rivers,Mark A. Roebke,Abraham M. Rutchick,Robert S. Ryan,Onur Sahin,Anondah R. Saide,Gillian M. Sandstrom,David Santos,David Santos,Rebecca Saxe,René Schlegelmilch,René Schlegelmilch,Kathleen Schmidt,Sabine Scholz,Larissa Seibel,Dylan Selterman,Samuel Shaki,William B. Simpson,H. Colleen Sinclair,Jeanine L. M. Skorinko,Agnieszka Slowik,Joel S. Snyder,Courtney K. Soderberg,Carina Sonnleitner,Nick Spencer,Jeffrey R. Spies,Sara Steegen,Stefan Stieger,Nina Strohminger,Gavin Brent Sullivan,Thomas Talhelm,Megan Tapia,Anniek M. te Dorsthorst,Manuela Thomae,Manuela Thomae,Sarah L. Thomas,Pia Tio,Frits Traets,Steve N.H. Tsang,Francis Tuerlinckx,Paul J. Turchan,Milan Valášek,Anna E. Van't Veer,Robbie C. M. van Aert,Marcel A.L.M. van Assen,Riet van Bork,Mathijs Van De Ven,Don van den Bergh,Marije van der Hulst,Roel van Dooren,Johnny van Doorn,Daan R. van Renswoude,Hedderik van Rijn,Wolf Vanpaemel,Alejandro Vásquez Echeverría,Melissa Vazquez,Natalia Vélez,Marieke Vermue,Mark Verschoor,Michelangelo Vianello,Martin Voracek,Gina Vuu,Eric-Jan Wagenmakers,Joanneke Weerdmeester,Ashlee Welsh,Erin C. Westgate,Joeri Wissink,Michael J. Wood,Andy T. Woods,Andy T. Woods,Emily M. Wright,Sining Wu,Marcel Zeelenberg,Kellylynn Zuni +290 more