The Alleged Crisis and the Illusion of Exact Replication
Wolfgang Stroebe,Fritz Strack +1 more
TLDR
It is proposed that for meaningful replications, attempts at reinstating the original circumstances are not sufficient and replicators must ascertain that conditions are realized that reflect the theoretical variable(s) manipulated (and/or measured) in the original study.Abstract:
There has been increasing criticism of the way psychologists conduct and analyze studies. These critiques as well as failures to replicate several high-profile studies have been used as justification to proclaim a “replication crisis” in psychology. Psychologists are encouraged to conduct more “exact” replications of published studies to assess the reproducibility of psychological research. This article argues that the alleged “crisis of replicability” is primarily due to an epistemological misunderstanding that emphasizes the phenomenon instead of its underlying mechanisms. As a consequence, a replicated phenomenon may not serve as a rigorous test of a theoretical hypothesis because identical operationalizations of variables in studies conducted at different times and with different subject populations might test different theoretical constructs. Therefore, we propose that for meaningful replications, attempts at reinstating the original circumstances are not sufficient. Instead, replicators must ascertain that conditions are realized that reflect the theoretical variable(s) manipulated (and/or measured) in the original study.read more
Citations
More filters
Journal ArticleDOI
The Chicago face database: A free stimulus set of faces and norming data
TL;DR: The Chicago Face Database is introduced, a free resource consisting of 158 high-resolution, standardized photographs of Black and White males and females between the ages of 18 and 40 years and extensive data about these targets and factors associated with researchers’ judgments of suitability.
Journal ArticleDOI
How to Do a Systematic Review: A Best Practice Guide for Conducting and Reporting Narrative Reviews, Meta-Analyses, and Meta-Syntheses.
TL;DR: It is argued that systematic reviews are a key methodology for clarifying whether and how research findings replicate and for explaining possible inconsistencies, and it is called for researchers to conduct systematic reviews to help elucidate whether there is a replication crisis.
Journal ArticleDOI
Is psychology suffering from a replication crisis? What does "failure to replicate" really mean?
TL;DR: This article suggests that so-called failures to replicate may not be failures at all, but rather are the result of low statistical power in single replication studies, and of failure to appreciate the need for multiple replications in order to have enough power to identify true effects.
Journal ArticleDOI
Don't fear 'fear conditioning': Methodological considerations for the design and analysis of studies on human fear acquisition, extinction, and return of fear
Tina B. Lonsdorf,Mareike M. Menz,Marta Andreatta,Miguel A. Fullana,Armita Golkar,Jan Haaker,Ivo Heitland,Andrea Hermann,Manuel Kuhn,Onno Kruse,Shira Meir Drexler,Ann Meulders,Frauke Nees,Andre Pittig,Jan Richter,Sonja Römer,Youssef Shiban,Anja Schmitz,Benjamin Straube,Bram Vervliet,Julia Wendt,Johanna M.P. Baas,Christian J. Merz +22 more
TL;DR: This compendium is intended to provide a basis for the development of a common procedural and terminology framework for the field of human fear conditioning and provides evidence‐based guidance for methodological decisions on study design, outcome measures, and analyses.
Journal ArticleDOI
Many Labs 2: Investigating Variation in Replicability Across Samples and Settings
Richard A. Klein,Michelangelo Vianello,Fred Hasselman,Byron G. Adams,Reginald B. Adams,Sinan Alper,Mark Aveyard,Jordan Axt,Mayowa T. Babalola,Štěpán Bahník,Rishtee Batra,Mihály Berkics,Michael J. Bernstein,Daniel R. Berry,Olga Bialobrzeska,Evans Dami Binan,Konrad Bocian,Mark J. Brandt,Robert Busching,Anna Cabak Rédei,Huajian Cai,Fanny Cambier,Katarzyna Cantarero,Cheryl L. Carmichael,Francisco Ceric,Jesse Chandler,Jen-Ho Chang,Armand Chatard,Eva E. Chen,Winnee Cheong,David C. Cicero,Sharon Coen,Jennifer A. Coleman,Brian Collisson,Morgan Conway,Katherine S. Corker,Paul G. Curran,Fiery Cushman,Zubairu K. Dagona,Ilker Dalgar,Anna Dalla Rosa,William E. Davis,Maaike J. de Bruijn,Leander De Schutter,Thierry Devos,Marieke de Vries,Marieke de Vries,Canay Doğulu,Nerisa Dozo,Kristin Nicole Dukes,Yarrow Dunham,Kevin Durrheim,Charles R. Ebersole,John E. Edlund,Anja Eller,Alexander S. English,Carolyn Finck,Natalia Frankowska,Miguel-Ángel Freyre,Mike Friedman,Elisa Maria Galliani,Joshua C. Gandi,Tanuka Ghoshal,Steffen R. Giessner,Tripat Gill,Timo Gnambs,Ángel Gómez,Roberto González,Jesse Graham,Jon Grahe,Ivan Grahek,Eva G. T. Green,Kakul Hai,Matthew Haigh,Elizabeth L. Haines,Michael P. Hall,Marie E. Heffernan,Joshua A. Hicks,Petr Houdek,Jeffrey R. Huntsinger,Ho Phi Huynh,Hans IJzerman,Yoel Inbar,Åse Innes-Ker,William Jiménez-Leal,Melissa-Sue John,Jennifer A. Joy-Gaba,Roza G. Kamiloğlu,Heather Barry Kappes,Serdar Karabati,Haruna Karick,Victor N. Keller,Anna Kende,Nicolas Kervyn,Goran Knežević,Carrie Kovacs,Lacy E. Krueger,German Kurapov,Jamie Kurtz,Daniel Lakens,Ljiljana B. Lazarević,Carmel A. Levitan,Neil A. Lewis,Samuel Lincoln Bezerra Lins,Nikolette P. Lipsey,Joy E. Losee,Esther Maassen,Angela T. Maitner,Winfrida Malingumu,Robyn K. Mallett,Satia A. Marotta,Janko Međedović,Fernando Mena-pacheco,Taciano L. Milfont,Wendy L. Morris,Sean C. Murphy,Andriy Myachykov,Nick Neave,Koen Ilja Neijenhuijs,Anthony J. Nelson,Félix Neto,Austin Lee Nichols,Aaron Ocampo,Susan L. O'Donnell,Haruka Oikawa,Masanori Oikawa,Elsie Ong,Gábor Orosz,Małgorzata Osowiecka,Grant Packard,Rolando Pérez-Sánchez,Boban Petrović,Ronaldo Pilati,Brad Pinter,Lysandra Podesta,Gabrielle Pogge,Monique Pollmann,Abraham M. Rutchick,Patricio Saavedra,Alexander K. Saeri,Erika Salomon,Kathleen Schmidt,Felix D. Schönbrodt,Maciej Sekerdej,David Sirlopú,Jeanine L. M. Skorinko,Michael A. Smith,Vanessa Smith-Castro,Karin C.H.J. Smolders,Agata Sobkow,Walter Sowden,Philipp Spachtholz,Manini Srivastava,Troy G. Steiner,Jeroen Stouten,Chris N. H. Street,Oskar K. Sundfelt,Stephanie Szeto,Ewa Szumowska,Andrew C. W. Tang,Norbert K. Tanzer,Morgan J. Tear,Jordan Theriault,Manuela Thomae,David Torres,Jakub Traczyk,Joshua M. Tybur,Adrienn Ujhelyi,Robbie C. M. van Aert,Marcel A.L.M. van Assen,Marije van der Hulst,Paul A. M. Van Lange,Anna van 't Veer,Alejandro Vásquez Echeverría,Leigh Ann Vaughn,Alexandra Vázquez,Luis Diego Vega,Catherine Verniers,Mark Verschoor,Ingrid P. J. Voermans,Marek A. Vranka,Cheryl Alyssa Welch,Aaron L. Wichman,Lisa A. Williams,Michael Wood,Julie A. Woodzicka,Marta Wrońska,Liane Young,John M. Zelenski,Zeng Zhijia,Brian A. Nosek +190 more
TL;DR: This paper conducted preregistered replications of 28 classic and contemporary published findings, with protocols that were peer reviewed in advance, to examine variation in effect magnitudes across samples and settings, and found that very little heterogeneity was attributable to the order in which the tasks were performed or whether the task were administered in lab versus online.
References
More filters
Book
A Theory of Cognitive Dissonance
TL;DR: Cognitive dissonance theory links actions and attitudes as discussed by the authors, which holds that dissonance is experienced whenever one cognition that a person holds follows from the opposite of at least one other cognition that the person holds.
Book
The Logic of Scientific Discovery
TL;DR: The Open Society and Its Enemies as discussed by the authors is regarded as one of Popper's most enduring books and contains insights and arguments that demand to be read to this day, as well as many of the ideas in the book.
Journal ArticleDOI
Stereotype threat and the intellectual test performance of African Americans
Claude M. Steele,Joshua Aronson +1 more
TL;DR: The role of stereotype vulnerability in the standardized test performance of ability-stigmatized groups is discussed and mere salience of the stereotype could impair Blacks' performance even when the test was not ability diagnostic.
Related Papers (5)
Estimating the reproducibility of psychological science
Alexander A. Aarts,Joanna E. Anderson,Christopher J. Anderson,Peter Raymond Attridge,Peter Raymond Attridge,Angela S. Attwood,Jordan Axt,Molly Babel,Štěpán Bahník,Erica Baranski,Michael Barnett-Cowan,Elizabeth Bartmess,Jennifer S. Beer,Raoul Bell,Heather Bentley,Leah Beyan,Grace Binion,Grace Binion,Denny Borsboom,Annick Bosch,Frank A. Bosco,Sara Bowman,Mark J. Brandt,Erin L Braswell,Hilmar Brohmer,Benjamin T. Brown,Kristina G. Brown,Jovita Brüning,Jovita Brüning,Ann Calhoun-Sauls,Shannon P. Callahan,Elizabeth Chagnon,Jesse Chandler,Jesse Chandler,Christopher R. Chartier,Felix Cheung,Felix Cheung,Cody D. Christopherson,Linda Cillessen,Russ Clay,Hayley M. D. Cleary,Mark D. Cloud,Michael Conn,Johanna Cohoon,Simon Columbus,Andreas Cordes,Giulio Costantini,Leslie Cramblet Alvarez,Ed Cremata,Jan Crusius,Jamie DeCoster,Michelle A. DeGaetano,Nicolás Delia Penna,Bobby Den Bezemer,Marie K. Deserno,Olivia Devitt,Laura Dewitte,David G. Dobolyi,Geneva T. Dodson,M. Brent Donnellan,Ryan Donohue,Rebecca A. Dore,Angela Rachael Dorrough,Angela Rachael Dorrough,Anna Dreber,Michelle Dugas,Elizabeth W. Dunn,Kayleigh E Easey,Sylvia Eboigbe,Casey Eggleston,Jo Embley,Sacha Epskamp,Timothy M. Errington,Vivien Estel,Frank J. Farach,Jenelle Feather,Anna Fedor,Belén Fernández-Castilla,Susann Fiedler,James G. Field,Stanka A. Fitneva,Taru Flagan,Amanda L. Forest,Eskil Forsell,Joshua D. Foster,Michael C. Frank,Rebecca S. Frazier,Heather M. Fuchs,Philip A. Gable,Jeff Galak,Elisa Maria Galliani,Anup Gampa,Sara García,Douglas Gazarian,Elizabeth Gilbert,Roger Giner-Sorolla,Andreas Glöckner,Andreas Glöckner,Lars Goellner,Jin X. Goh,Rebecca M. Goldberg,Patrick T. Goodbourn,Shauna Gordon-McKeon,Bryan Gorges,Jessie Gorges,Justin Goss,Jesse Graham,James A. Grange,Jeremy R. Gray,Chris H.J. Hartgerink,Joshua K. Hartshorne,Fred Hasselman,Timothy Hayes,Emma Heikensten,Felix Henninger,Felix Henninger,John Hodsoll,Taylor Holubar,Gea Hoogendoorn,Denise J. Humphries,Cathy On-Ying Hung,Nathali Immelman,Vanessa C. Irsik,Georg Jahn,Frank Jäkel,Marc Jekel,Magnus Johannesson,Larissa Gabrielle Johnson,David J. Johnson,Kate M. Johnson,William J. Johnston,Kai J. Jonas,Jennifer A. Joy-Gaba,Heather Barry Kappes,Kim Kelso,Mallory C. Kidwell,Seung K. Kim,Matthew W. Kirkhart,Bennett Kleinberg,Bennett Kleinberg,Goran Knežević,Franziska Maria Kolorz,Jolanda J. Kossakowski,Robert Krause,Job Krijnen,Tim Kuhlmann,Yoram K. Kunkels,Megan M. Kyc,Calvin K. Lai,Aamir Laique,Daniel Lakens,Kristin A. Lane,Bethany Lassetter,Ljiljana B. Lazarević,Etienne P. Le Bel,Key Jung Lee,Minha Lee,Kristi M. Lemm,Carmel A. Levitan,Melissa Lewis,Lin Lin,Stephanie C. Lin,Matthias Lippold,Darren Loureiro,Ilse Luteijn,Sean P. Mackinnon,Heather N. Mainard,Denise C. Marigold,Daniel P. Martin,Tylar Martinez,E. J. Masicampo,Joshua J. Matacotta,Maya B. Mathur,Michael May,Michael May,Nicole Mechin,Pranjal H. Mehta,Johannes M. Meixner,Johannes M. Meixner,Alissa Melinger,Jeremy K. Miller,Mallorie Miller,Katherine Moore,Katherine Moore,Marcus Möschl,Matt Motyl,Stephanie M. Müller,Marcus R. Munafò,Koen Ilja Neijenhuijs,Taylor Nervi,Gandalf Nicolas,Gustav Nilsonne,Gustav Nilsonne,Brian A. Nosek,Brian A. Nosek,Michèle B. Nuijten,Catherine Olsson,Catherine Olsson,Colleen Osborne,Lutz Ostkamp,Misha Pavel,Ian S. Penton-Voak,Olivia Perna,Cyril Pernet,Marco Perugini,R. Nathan Pipitone,Michael C. Pitts,Franziska Plessow,Franziska Plessow,Jason M. Prenoveau,Rima-Maria Rahal,Rima-Maria Rahal,Kate A. Ratliff,David Reinhard,Frank Renkewitz,Ashley A. Ricker,Anastasia E. Rigney,Andrew M Rivers,Mark A. Roebke,Abraham M. Rutchick,Robert S. Ryan,Onur Sahin,Anondah R. Saide,Gillian M. Sandstrom,David Santos,David Santos,Rebecca Saxe,René Schlegelmilch,René Schlegelmilch,Kathleen Schmidt,Sabine Scholz,Larissa Seibel,Dylan Selterman,Samuel Shaki,William B. Simpson,H. Colleen Sinclair,Jeanine L. M. Skorinko,Agnieszka Slowik,Joel S. Snyder,Courtney K. Soderberg,Carina Sonnleitner,Nick Spencer,Jeffrey R. Spies,Sara Steegen,Stefan Stieger,Nina Strohminger,Gavin Brent Sullivan,Thomas Talhelm,Megan Tapia,Anniek M. te Dorsthorst,Manuela Thomae,Manuela Thomae,Sarah L. Thomas,Pia Tio,Frits Traets,Steve N.H. Tsang,Francis Tuerlinckx,Paul J. Turchan,Milan Valášek,Anna E. Van't Veer,Robbie C. M. van Aert,Marcel A.L.M. van Assen,Riet van Bork,Mathijs Van De Ven,Don van den Bergh,Marije van der Hulst,Roel van Dooren,Johnny van Doorn,Daan R. van Renswoude,Hedderik van Rijn,Wolf Vanpaemel,Alejandro Vásquez Echeverría,Melissa Vazquez,Natalia Vélez,Marieke Vermue,Mark Verschoor,Michelangelo Vianello,Martin Voracek,Gina Vuu,Eric-Jan Wagenmakers,Joanneke Weerdmeester,Ashlee Welsh,Erin C. Westgate,Joeri Wissink,Michael J. Wood,Andy T. Woods,Andy T. Woods,Emily M. Wright,Sining Wu,Marcel Zeelenberg,Kellylynn Zuni +290 more
False-Positive Psychology: Undisclosed Flexibility in Data Collection and Analysis Allows Presenting Anything as Significant
Measuring the Prevalence of Questionable Research Practices With Incentives for Truth Telling
Editors' Introduction to the Special Section on Replicability in Psychological Science: A Crisis of Confidence?
Investigating variation in replicability: A “Many Labs” replication project
Richard A. Klein,Kate A. Ratliff,Michelangelo Vianello,Reginald B. Adams,Štěpán Bahník,Michael J. Bernstein,Konrad Bocian,Mark J. Brandt,Beach Brooks,Claudia Chloe Brumbaugh,Zeynep Cemalcilar,Jesse Chandler,Winnee Cheong,William E. Davis,Thierry Devos,Matthew Eisner,Natalia Frankowska,David Furrow,Elisa Maria Galliani,Fred Hasselman,Joshua A. Hicks,James Hovermale,S. Jane Hunt,Jeffrey R. Huntsinger,Hans IJzerman,Melissa-Sue John,Jennifer A. Joy-Gaba,Heather Barry Kappes,Lacy E. Krueger,Jaime L. Kurtz,Carmel A. Levitan,Robyn K. Mallett,Wendy L. Morris,Anthony J. Nelson,Jason A. Nier,Grant Packard,Ronaldo Pilati,Abraham M. Rutchick,Kathleen Schmidt,Jeanine L. M. Skorinko,Robert W. Smith,Troy G. Steiner,Justin Storbeck,Lyn M. Van Swol,Donna Thompson,A. E. van ‘t Veer,Leigh Ann Vaughn,Marek A. Vranka,Aaron L. Wichman,Julie A. Woodzicka,Brian A. Nosek +50 more