Replication, Communication, and the Population Dynamics of Scientific Discovery
Reads0
Chats0
TLDR
A mathematical model of scientific discovery that combines hypothesis formation, replication, publication bias, and variation in research quality is developed and it is found that communication of negative replications may aid true discovery even when attempts to replicate have diminished power.Abstract:
Many published research results are false (Ioannidis, 2005), and controversy continues over the roles of replication and publication policy in improving the reliability of research. Addressing these problems is frustrated by the lack of a formal framework that jointly represents hypothesis formation, replication, publication bias, and variation in research quality. We develop a mathematical model of scientific discovery that combines all of these elements. This model provides both a dynamic model of research as well as a formal framework for reasoning about the normative structure of science. We show that replication may serve as a ratchet that gradually separates true hypotheses from false, but the same factors that make initial findings unreliable also make replications unreliable. The most important factors in improving the reliability of research are the rate of false positives and the base rate of true hypotheses, and we offer suggestions for addressing each. Our results also bring clarity to verbal debates about the communication of research. Surprisingly, publication bias is not always an obstacle, but instead may have positive impacts—suppression of negative novel findings is often beneficial. We also find that communication of negative replications may aid true discovery even when attempts to replicate have diminished power. The model speaks constructively to ongoing debates about the design and conduct of science, focusing analysis and discussion on precise, internally consistent models, as well as highlighting the importance of population dynamics.read more
Citations
More filters
Journal ArticleDOI
The natural selection of bad science.
TL;DR: In this paper, the authors present a 60-year meta-analysis of statistical power in the behavioural sciences and show that power has not improved despite repeated demonstrations of the necessity of increasing power.
Journal ArticleDOI
The Natural Selection of Bad Science
TL;DR: A 60-year meta-analysis of statistical power in the behavioural sciences is presented and it is shown that power has not improved despite repeated demonstrations of the necessity of increasing power, and that replication slows but does not stop the process of methodological deterioration.
Journal ArticleDOI
“Fake News” Is Not Simply False Information: A Concept Explication and Taxonomy of Online Content:
TL;DR: An explication of “fake news” that, as a concept, has ballooned to include more than simply false information, with partisans weaponizing it to cast aspersions on the veracity of claims made by those who are politically opposed to them is conducted.
Journal ArticleDOI
Publication bias and the canonization of false facts.
TL;DR: It is found that unless a sufficient fraction of negative results are published, false claims frequently can become canonized as fact and true and false claims would be more readily distinguished.
Book ChapterDOI
Models Are Stupid, and We Need More of Them
TL;DR: The problem with most verbal models is that there are many ways to specify the parts and relationships of a system that are consistent with such a model, and this has the potential disadvantage of making formal models appear stupid as discussed by the authors.
References
More filters
Journal ArticleDOI
The file drawer problem and tolerance for null results
TL;DR: Quantitative procedures for computing the tolerance for filed and future null results are reported and illustrated, and the implications are discussed.
Journal ArticleDOI
Power failure: why small sample size undermines the reliability of neuroscience
Katherine S. Button,John P. A. Ioannidis,Claire Mokrysz,Brian A. Nosek,Jonathan Flint,Emma S J Robinson,Marcus R. Munafò +6 more
TL;DR: It is shown that the average statistical power of studies in the neurosciences is very low, and the consequences include overestimates of effect size and low reproducibility of results.
Why Most Published Research Findings Are False
TL;DR: In this paper, the authors discuss the implications of these problems for the conduct and interpretation of research and suggest that claimed research findings may often be simply accurate measures of the prevailing bias.
Journal ArticleDOI
False-Positive Psychology: Undisclosed Flexibility in Data Collection and Analysis Allows Presenting Anything as Significant
TL;DR: It is shown that despite empirical psychologists’ nominal endorsement of a low rate of false-positive findings, flexibility in data collection, analysis, and reporting dramatically increases actual false- positive rates, and a simple, low-cost, and straightforwardly effective disclosure-based solution is suggested.
Book
Conjectures and Refutations: The Growth of Scientific Knowledge
TL;DR: A collection of classic essays written throughout Popper's illustrious career, expounding and defending his 'fallibilist' theory of knowledge and scientific discovery.
Related Papers (5)
False-Positive Psychology: Undisclosed Flexibility in Data Collection and Analysis Allows Presenting Anything as Significant
Estimating the reproducibility of psychological science
Alexander A. Aarts,Joanna E. Anderson,Christopher J. Anderson,Peter Raymond Attridge,Peter Raymond Attridge,Angela S. Attwood,Jordan Axt,Molly Babel,Štěpán Bahník,Erica Baranski,Michael Barnett-Cowan,Elizabeth Bartmess,Jennifer S. Beer,Raoul Bell,Heather Bentley,Leah Beyan,Grace Binion,Grace Binion,Denny Borsboom,Annick Bosch,Frank A. Bosco,Sara Bowman,Mark J. Brandt,Erin L Braswell,Hilmar Brohmer,Benjamin T. Brown,Kristina G. Brown,Jovita Brüning,Jovita Brüning,Ann Calhoun-Sauls,Shannon P. Callahan,Elizabeth Chagnon,Jesse Chandler,Jesse Chandler,Christopher R. Chartier,Felix Cheung,Felix Cheung,Cody D. Christopherson,Linda Cillessen,Russ Clay,Hayley M. D. Cleary,Mark D. Cloud,Michael Conn,Johanna Cohoon,Simon Columbus,Andreas Cordes,Giulio Costantini,Leslie Cramblet Alvarez,Ed Cremata,Jan Crusius,Jamie DeCoster,Michelle A. DeGaetano,Nicolás Delia Penna,Bobby Den Bezemer,Marie K. Deserno,Olivia Devitt,Laura Dewitte,David G. Dobolyi,Geneva T. Dodson,M. Brent Donnellan,Ryan Donohue,Rebecca A. Dore,Angela Rachael Dorrough,Angela Rachael Dorrough,Anna Dreber,Michelle Dugas,Elizabeth W. Dunn,Kayleigh E Easey,Sylvia Eboigbe,Casey Eggleston,Jo Embley,Sacha Epskamp,Timothy M. Errington,Vivien Estel,Frank J. Farach,Jenelle Feather,Anna Fedor,Belén Fernández-Castilla,Susann Fiedler,James G. Field,Stanka A. Fitneva,Taru Flagan,Amanda L. Forest,Eskil Forsell,Joshua D. Foster,Michael C. Frank,Rebecca S. Frazier,Heather M. Fuchs,Philip A. Gable,Jeff Galak,Elisa Maria Galliani,Anup Gampa,Sara García,Douglas Gazarian,Elizabeth Gilbert,Roger Giner-Sorolla,Andreas Glöckner,Andreas Glöckner,Lars Goellner,Jin X. Goh,Rebecca M. Goldberg,Patrick T. Goodbourn,Shauna Gordon-McKeon,Bryan Gorges,Jessie Gorges,Justin Goss,Jesse Graham,James A. Grange,Jeremy R. Gray,Chris H.J. Hartgerink,Joshua K. Hartshorne,Fred Hasselman,Timothy Hayes,Emma Heikensten,Felix Henninger,Felix Henninger,John Hodsoll,Taylor Holubar,Gea Hoogendoorn,Denise J. Humphries,Cathy On-Ying Hung,Nathali Immelman,Vanessa C. Irsik,Georg Jahn,Frank Jäkel,Marc Jekel,Magnus Johannesson,Larissa Gabrielle Johnson,David J. Johnson,Kate M. Johnson,William J. Johnston,Kai J. Jonas,Jennifer A. Joy-Gaba,Heather Barry Kappes,Kim Kelso,Mallory C. Kidwell,Seung K. Kim,Matthew W. Kirkhart,Bennett Kleinberg,Bennett Kleinberg,Goran Knežević,Franziska Maria Kolorz,Jolanda J. Kossakowski,Robert Krause,Job Krijnen,Tim Kuhlmann,Yoram K. Kunkels,Megan M. Kyc,Calvin K. Lai,Aamir Laique,Daniel Lakens,Kristin A. Lane,Bethany Lassetter,Ljiljana B. Lazarević,Etienne P. Le Bel,Key Jung Lee,Minha Lee,Kristi M. Lemm,Carmel A. Levitan,Melissa Lewis,Lin Lin,Stephanie C. Lin,Matthias Lippold,Darren Loureiro,Ilse Luteijn,Sean P. Mackinnon,Heather N. Mainard,Denise C. Marigold,Daniel P. Martin,Tylar Martinez,E. J. Masicampo,Joshua J. Matacotta,Maya B. Mathur,Michael May,Michael May,Nicole Mechin,Pranjal H. Mehta,Johannes M. Meixner,Johannes M. Meixner,Alissa Melinger,Jeremy K. Miller,Mallorie Miller,Katherine Moore,Katherine Moore,Marcus Möschl,Matt Motyl,Stephanie M. Müller,Marcus R. Munafò,Koen Ilja Neijenhuijs,Taylor Nervi,Gandalf Nicolas,Gustav Nilsonne,Gustav Nilsonne,Brian A. Nosek,Brian A. Nosek,Michèle B. Nuijten,Catherine Olsson,Catherine Olsson,Colleen Osborne,Lutz Ostkamp,Misha Pavel,Ian S. Penton-Voak,Olivia Perna,Cyril Pernet,Marco Perugini,R. Nathan Pipitone,Michael C. Pitts,Franziska Plessow,Franziska Plessow,Jason M. Prenoveau,Rima-Maria Rahal,Rima-Maria Rahal,Kate A. Ratliff,David Reinhard,Frank Renkewitz,Ashley A. Ricker,Anastasia E. Rigney,Andrew M Rivers,Mark A. Roebke,Abraham M. Rutchick,Robert S. Ryan,Onur Sahin,Anondah R. Saide,Gillian M. Sandstrom,David Santos,David Santos,Rebecca Saxe,René Schlegelmilch,René Schlegelmilch,Kathleen Schmidt,Sabine Scholz,Larissa Seibel,Dylan Selterman,Samuel Shaki,William B. Simpson,H. Colleen Sinclair,Jeanine L. M. Skorinko,Agnieszka Slowik,Joel S. Snyder,Courtney K. Soderberg,Carina Sonnleitner,Nick Spencer,Jeffrey R. Spies,Sara Steegen,Stefan Stieger,Nina Strohminger,Gavin Brent Sullivan,Thomas Talhelm,Megan Tapia,Anniek M. te Dorsthorst,Manuela Thomae,Manuela Thomae,Sarah L. Thomas,Pia Tio,Frits Traets,Steve N.H. Tsang,Francis Tuerlinckx,Paul J. Turchan,Milan Valášek,Anna E. Van't Veer,Robbie C. M. van Aert,Marcel A.L.M. van Assen,Riet van Bork,Mathijs Van De Ven,Don van den Bergh,Marije van der Hulst,Roel van Dooren,Johnny van Doorn,Daan R. van Renswoude,Hedderik van Rijn,Wolf Vanpaemel,Alejandro Vásquez Echeverría,Melissa Vazquez,Natalia Vélez,Marieke Vermue,Mark Verschoor,Michelangelo Vianello,Martin Voracek,Gina Vuu,Eric-Jan Wagenmakers,Joanneke Weerdmeester,Ashlee Welsh,Erin C. Westgate,Joeri Wissink,Michael J. Wood,Andy T. Woods,Andy T. Woods,Emily M. Wright,Sining Wu,Marcel Zeelenberg,Kellylynn Zuni +290 more