Open Access
Statistical Analyses for Studying Replication: Meta-Analytic Perspectives.
Larry V. Hedges,Jacob M. Schauer +1 more
- Vol. 24, Iss: 5, pp 557-570
Reads0
Chats0
TLDR
The authors provide alternative conceptual frameworks that lead to different statistical analyses to test hypotheses about replication, including whether the burden of proof is placed on replication or nonreplication, whether replication is exact or allows for a small amount of "negligible heterogeneity," and whether the studies observed are assumed to be fixed (constituting the entire body of relevant evidence) or are a sample from a universe of possibly relevant studies.Abstract:
Formal empirical assessments of replication have recently become more prominent in several areas of science, including psychology. These assessments have used different statistical approaches to determine if a finding has been replicated. The purpose of this article is to provide several alternative conceptual frameworks that lead to different statistical analyses to test hypotheses about replication. All of these analyses are based on statistical methods used in meta-analysis. The differences among the methods described involve whether the burden of proof is placed on replication or nonreplication, whether replication is exact or allows for a small amount of "negligible heterogeneity," and whether the studies observed are assumed to be fixed (constituting the entire body of relevant evidence) or are a sample from a universe of possibly relevant studies. The statistical power of each of these tests is computed and shown to be low in many cases, raising issues of the interpretability of tests for replication. (PsycINFO Database Record (c) 2019 APA, all rights reserved).read more
Citations
More filters
Preprint of Too good to be false: Nonsignificant results revisited
TL;DR: This article examined evidence for false negatives in nonsignificant results in three different ways and concluded that false negatives deserve more attention in the current debate on statistical practices in psychology, and they also proposed the adapted Fisher test to detect the presence of at least one false negative in a set of statistically nonsignificantly results.
Design-Based Approaches to Causal Replication Studies
TL;DR: The paper describes research designs for replication and demonstrates how multiple designs may be combined in systematic replication efforts, as well as how diagnostic measures may be used to assess the extent to which CRF assumptions are met in field settings.
Journal ArticleDOI
On the bias of complete‐ and shifting‐case meta‐regressions with missing covariates
TL;DR: In this paper , the authors study the potential magnitude of that bias assuming a log-linear model of missingness and find that the bias can be substantial, as large as Cohen's d = 0.4-0.8 depending on the missingness mechanism.
Journal ArticleDOI
Beyond Random Effects: When Small-Study Findings Are More Heterogeneous
TL;DR: In this article , meta-regression methods are introduced that identify whether the magnitude of heterogeneity across study findings is correlated with their standard errors and find that small-sample studies typically have higher heterogeneity.
Peer Review
Replicability Across Multiple Studies
Marina Bogomolov,Ruth Heller +1 more
TL;DR: A selective overview of analyses that can be carried out towards establishing replicability of the scientific findings is provided and methods for the setting where a single outcome is examined in multiple studies are described.
References
More filters
Book
Cochrane Handbook for Systematic Reviews of Interventions
Julian P T Higgins,Sally Green +1 more
TL;DR: The Cochrane Handbook for Systematic Reviews of Interventions is the official document that describes in detail the process of preparing and maintaining Cochrane systematic reviews on the effects of healthcare interventions.
Journal ArticleDOI
Statistical Methods for Meta-Analysis.
TL;DR: In this paper, the authors present a model for estimating the effect size from a series of experiments using a fixed effect model and a general linear model, and combine these two models to estimate the effect magnitude.
Journal ArticleDOI
Estimating the reproducibility of psychological science
Alexander A. Aarts,Joanna E. Anderson,Christopher J. Anderson,Peter Raymond Attridge,Peter Raymond Attridge,Angela S. Attwood,Jordan Axt,Molly Babel,Štěpán Bahník,Erica Baranski,Michael Barnett-Cowan,Elizabeth Bartmess,Jennifer S. Beer,Raoul Bell,Heather Bentley,Leah Beyan,Grace Binion,Grace Binion,Denny Borsboom,Annick Bosch,Frank A. Bosco,Sara Bowman,Mark J. Brandt,Erin L Braswell,Hilmar Brohmer,Benjamin T. Brown,Kristina G. Brown,Jovita Brüning,Jovita Brüning,Ann Calhoun-Sauls,Shannon P. Callahan,Elizabeth Chagnon,Jesse Chandler,Jesse Chandler,Christopher R. Chartier,Felix Cheung,Felix Cheung,Cody D. Christopherson,Linda Cillessen,Russ Clay,Hayley M. D. Cleary,Mark D. Cloud,Michael Conn,Johanna Cohoon,Simon Columbus,Andreas Cordes,Giulio Costantini,Leslie Cramblet Alvarez,Ed Cremata,Jan Crusius,Jamie DeCoster,Michelle A. DeGaetano,Nicolás Delia Penna,Bobby Den Bezemer,Marie K. Deserno,Olivia Devitt,Laura Dewitte,David G. Dobolyi,Geneva T. Dodson,M. Brent Donnellan,Ryan Donohue,Rebecca A. Dore,Angela Rachael Dorrough,Angela Rachael Dorrough,Anna Dreber,Michelle Dugas,Elizabeth W. Dunn,Kayleigh E Easey,Sylvia Eboigbe,Casey Eggleston,Jo Embley,Sacha Epskamp,Timothy M. Errington,Vivien Estel,Frank J. Farach,Jenelle Feather,Anna Fedor,Belén Fernández-Castilla,Susann Fiedler,James G. Field,Stanka A. Fitneva,Taru Flagan,Amanda L. Forest,Eskil Forsell,Joshua D. Foster,Michael C. Frank,Rebecca S. Frazier,Heather M. Fuchs,Philip A. Gable,Jeff Galak,Elisa Maria Galliani,Anup Gampa,Sara García,Douglas Gazarian,Elizabeth Gilbert,Roger Giner-Sorolla,Andreas Glöckner,Andreas Glöckner,Lars Goellner,Jin X. Goh,Rebecca M. Goldberg,Patrick T. Goodbourn,Shauna Gordon-McKeon,Bryan Gorges,Jessie Gorges,Justin Goss,Jesse Graham,James A. Grange,Jeremy R. Gray,Chris H.J. Hartgerink,Joshua K. Hartshorne,Fred Hasselman,Timothy Hayes,Emma Heikensten,Felix Henninger,Felix Henninger,John Hodsoll,Taylor Holubar,Gea Hoogendoorn,Denise J. Humphries,Cathy On-Ying Hung,Nathali Immelman,Vanessa C. Irsik,Georg Jahn,Frank Jäkel,Marc Jekel,Magnus Johannesson,Larissa Gabrielle Johnson,David J. Johnson,Kate M. Johnson,William J. Johnston,Kai J. Jonas,Jennifer A. Joy-Gaba,Heather Barry Kappes,Kim Kelso,Mallory C. Kidwell,Seung K. Kim,Matthew W. Kirkhart,Bennett Kleinberg,Bennett Kleinberg,Goran Knežević,Franziska Maria Kolorz,Jolanda J. Kossakowski,Robert Krause,Job Krijnen,Tim Kuhlmann,Yoram K. Kunkels,Megan M. Kyc,Calvin K. Lai,Aamir Laique,Daniel Lakens,Kristin A. Lane,Bethany Lassetter,Ljiljana B. Lazarević,Etienne P. Le Bel,Key Jung Lee,Minha Lee,Kristi M. Lemm,Carmel A. Levitan,Melissa Lewis,Lin Lin,Stephanie C. Lin,Matthias Lippold,Darren Loureiro,Ilse Luteijn,Sean P. Mackinnon,Heather N. Mainard,Denise C. Marigold,Daniel P. Martin,Tylar Martinez,E. J. Masicampo,Joshua J. Matacotta,Maya B. Mathur,Michael May,Michael May,Nicole Mechin,Pranjal H. Mehta,Johannes M. Meixner,Johannes M. Meixner,Alissa Melinger,Jeremy K. Miller,Mallorie Miller,Katherine Moore,Katherine Moore,Marcus Möschl,Matt Motyl,Stephanie M. Müller,Marcus R. Munafò,Koen Ilja Neijenhuijs,Taylor Nervi,Gandalf Nicolas,Gustav Nilsonne,Gustav Nilsonne,Brian A. Nosek,Brian A. Nosek,Michèle B. Nuijten,Catherine Olsson,Catherine Olsson,Colleen Osborne,Lutz Ostkamp,Misha Pavel,Ian S. Penton-Voak,Olivia Perna,Cyril Pernet,Marco Perugini,R. Nathan Pipitone,Michael C. Pitts,Franziska Plessow,Franziska Plessow,Jason M. Prenoveau,Rima-Maria Rahal,Rima-Maria Rahal,Kate A. Ratliff,David Reinhard,Frank Renkewitz,Ashley A. Ricker,Anastasia E. Rigney,Andrew M Rivers,Mark A. Roebke,Abraham M. Rutchick,Robert S. Ryan,Onur Sahin,Anondah R. Saide,Gillian M. Sandstrom,David Santos,David Santos,Rebecca Saxe,René Schlegelmilch,René Schlegelmilch,Kathleen Schmidt,Sabine Scholz,Larissa Seibel,Dylan Selterman,Samuel Shaki,William B. Simpson,H. Colleen Sinclair,Jeanine L. M. Skorinko,Agnieszka Slowik,Joel S. Snyder,Courtney K. Soderberg,Carina Sonnleitner,Nick Spencer,Jeffrey R. Spies,Sara Steegen,Stefan Stieger,Nina Strohminger,Gavin Brent Sullivan,Thomas Talhelm,Megan Tapia,Anniek M. te Dorsthorst,Manuela Thomae,Manuela Thomae,Sarah L. Thomas,Pia Tio,Frits Traets,Steve N.H. Tsang,Francis Tuerlinckx,Paul J. Turchan,Milan Valášek,Anna E. Van't Veer,Robbie C. M. van Aert,Marcel A.L.M. van Assen,Riet van Bork,Mathijs Van De Ven,Don van den Bergh,Marije van der Hulst,Roel van Dooren,Johnny van Doorn,Daan R. van Renswoude,Hedderik van Rijn,Wolf Vanpaemel,Alejandro Vásquez Echeverría,Melissa Vazquez,Natalia Vélez,Marieke Vermue,Mark Verschoor,Michelangelo Vianello,Martin Voracek,Gina Vuu,Eric-Jan Wagenmakers,Joanneke Weerdmeester,Ashlee Welsh,Erin C. Westgate,Joeri Wissink,Michael J. Wood,Andy T. Woods,Andy T. Woods,Emily M. Wright,Sining Wu,Marcel Zeelenberg,Kellylynn Zuni +290 more
TL;DR: A large-scale assessment suggests that experimental reproducibility in psychology leaves a lot to be desired, and correlational tests suggest that replication success was better predicted by the strength of original evidence than by characteristics of the original and replication teams.
Journal ArticleDOI
Review of particle properties.
K. Hagiwara,Ken Ichi Hikasa,Koji Nakamura,Masaharu Tanabashi,M. Aguilar-Benitez,Claude Amsler,R. M. Barnett,Patricia R. Burchat,C. D. Carone,C. Caso,G. Conforto,Olav Dahl,Michael Doser,Semen Eidelman,Jonathan L. Feng,L. K. Gibbons,Maury Goodman,Christoph Grab,D. E. Groom,Atul Gurtu,Atul Gurtu,K. G. Hayes,J. J. Herna`ndez-Rey,K. Honscheid,Christopher Kolda,Michelangelo L. Mangano,David Manley,Aneesh V. Manohar,John March-Russell,Alberto Masoni,Ramon Miquel,Klaus Mönig,Hitoshi Murayama,Hitoshi Murayama,S. Sánchez Navas,Keith A. Olive,Luc Pape,C. Patrignani,A. Piepke,Matts Roos,John Terning,Nils A. Tornqvist,T. G. Trippe,Petr Vogel,C. G. Wohl,Ron L. Workman,W-M. Yao,B. Armstrong,P. S. Gee,K. S. Lugovsky,S. B. Lugovsky,V. S. Lugovsky,Marina Artuso,D. Asner,K. S. Babu,E. L. Barberio,Marco Battaglia,H. Bichsel,O. Biebel,Philippe Bloch,Robert N. Cahn,Ariella Cattai,R. S. Chivukula,R. Cousins,G. A. Cowan,Thibault Damour,K. Desler,R. J. Donahue,D. A. Edwards,Victor Daniel Elvira,Jens Erler,V. V. Ezhela,A Fassò,W. Fetscher,Brian D. Fields,B. Foster,Daniel Froidevaux,Masataka Fukugita,Thomas K. Gaisser,L. Garren,H.-J. Gerber,Frederick J. Gilman,Howard E. Haber,C. A. Hagmann,J.L. Hewett,Ian Hinchliffe,Craig J. Hogan,G. Höhler,P. Igo-Kemenes,John David Jackson,Kurtis F Johnson,D. Karlen,B. Kayser,S. R. Klein,Konrad Kleinknecht,I.G. Knowles,P. Kreitz,Yu V. Kuyanov,R. Landua,Paul Langacker,L. S. Littenberg,Alan D. Martin,Tatsuya Nakada,Tatsuya Nakada,Meenakshi Narain,Paolo Nason,John A. Peacock,Helen R. Quinn,Stuart Raby,Georg G. Raffelt,E. A. Razuvaev,B. Renk,L. Rolandi,Michael T Ronan,L.J. Rosenberg,Christopher T. Sachrajda,A. I. Sanda,Subir Sarkar,Michael Schmitt,O. Schneider,Douglas Scott,W. G. Seligman,Michael H. Shaevitz,Torbjörn Sjöstrand,George F. Smoot,Stefan M Spanier,H. Spieler,N. J. C. Spooner,Mark Srednicki,A. Stahl,Todor Stanev,M. Suzuki,N. P. Tkachenko,German Valencia,K. van Bibber,Manuella Vincter,D. R. Ward,Bryan R. Webber,M R Whalley,Lincoln Wolfenstein,J. Womersley,C. L. Woody,O. V. Zenin +142 more
TL;DR: This biennial Review summarizes much of Particle Physics using data from previous editions, plus 2205 new measurements from 667 papers, and features expanded coverage of CP violation in B mesons and of neutrino oscillations.
Book
Methods of Meta-Analysis: Correcting Error and Bias in Research Findings
John E. Hunter,Frank L. Schmidt +1 more
TL;DR: In this article, the authors present a meta-analysis of Artifact Distributions and their impact on study outcomes. But they focus mainly on the second-order sampling error and related issues.
Related Papers (5)
Statistical analyses for studying replication: Meta-analytic perspectives.
Larry V. Hedges,Jacob M. Schauer +1 more