1,500 scientists lift the lid on reproducibility
About: This article is published in Nature.The article was published on 2016-05-26 and is currently open access. It has received 2609 citations till now. The article focuses on the topics: Lift (data mining).
Citations
More filters
TL;DR: This work argues for the adoption of measures to optimize key elements of the scientific process: methods, reporting and dissemination, reproducibility, evaluation and incentives, in the hope that this will facilitate action toward improving the transparency, reproducible and efficiency of scientific research.
Abstract: Improving the reliability and efficiency of scientific research will increase the credibility of the published scientific literature and accelerate discovery. Here we argue for the adoption of measures to optimize key elements of the scientific process: methods, reporting and dissemination, reproducibility, evaluation and incentives. There is some evidence from both simulations and empirical studies supporting the likely effectiveness of these measures, but their broad adoption by researchers, institutions, funders and journals will require iterative evaluation and improvement. We discuss the goals of these measures, and how they can be implemented, in the hope that this will facilitate action toward improving the transparency, reproducibility and efficiency of scientific research.
1,951 citations
California Institute of Technology1, Stockholm School of Economics2, University of Innsbruck3, National University of Singapore4, University of Pennsylvania5, Center for Open Science6, University of Virginia7, Harvard University8, Ludwig Maximilian University of Munich9, University of Amsterdam10, Harbin Institute of Technology11
TL;DR: It is found that peer beliefs of replicability are strongly related to replicable, suggesting that the research community could predict which results would replicate and that failures to replicate were not the result of chance alone.
Abstract: Being able to replicate scientific findings is crucial for scientific progress. We replicate 21 systematically selected experimental studies in the social sciences published in Nature and Science between 2010 and 2015. The replications follow analysis plans reviewed by the original authors and pre-registered prior to the replications. The replications are high powered, with sample sizes on average about five times higher than in the original studies. We find a significant effect in the same direction as the original study for 13 (62%) studies, and the effect size of the replications is on average about 50% of the original effect size. Replicability varies between 12 (57%) and 14 (67%) studies for complementary replicability indicators. Consistent with these results, the estimated true-positive rate is 67% in a Bayesian analysis. The relative effect size of true positives is estimated to be 71%, suggesting that both false positives and inflated effect sizes of true positives contribute to imperfect reproducibility. Furthermore, we find that peer beliefs of replicability are strongly related to replicability, suggesting that the research community could predict which results would replicate and that failures to replicate were not the result of chance alone.
759 citations
TL;DR: The present Bioconda, a distribution of bioinformatics software for the lightweight, multi-platform and language-agnostic package manager Conda, improves analysis reproducibility by allowing users to define isolated environments with defined software versions.
Abstract: We present Bioconda (https://bioconda.github.io), a distribution of bioinformatics software for the lightweight, multi-platform and language-agnostic package manager Conda. Currently, Bioconda offers a collection of over 3000 software packages, which is continuously maintained, updated, and extended by a growing global community of more than 200 contributors. Bioconda improves analysis reproducibility by allowing users to define isolated environments with defined software versions, all of which are easily installed and managed without administrative privileges.
699 citations
TL;DR: The nf-core framework is introduced as a means for the development of collaborative, peerreviewed, best-practice analysis pipelines that can be used across all institutions and research facilities and introduces a higher degree of portability as compared to custom in-house scripts.
Abstract: To the Editor — The standardization, portability and reproducibility of analysis pipelines are key issues within the bioinformatics community. Most bioinformatics pipelines are designed for use on-premises; as a result, the associated software dependencies and execution logic are likely to be tightly coupled with proprietary computing environments. This can make it difficult or even impossible for others to reproduce the ensuing results, which is a fundamental requirement for the validation of scientific findings. Here, we introduce the nf-core framework as a means for the development of collaborative, peerreviewed, best-practice analysis pipelines (Fig. 1). All nf-core pipelines are written in Nextflow and so inherit the ability to be executed on most computational infrastructures, as well as having native support for container technologies such as Docker and Singularity. The nf-core community (Supplementary Fig. 1) has developed a suite of tools that automate pipeline creation, testing, deployment and synchronization. Our goal is to provide a framework for high-quality bioinformatics pipelines that can be used across all institutions and research facilities. Being able to reproduce scientific results is the central tenet of the scientific method. However, moving toward FAIR (findable, accessible, interoperable and reusable) research methods1 in data-driven science is complex2,3. Central repositories, such as bio. tools4, omictools5 and the Galaxy toolshed6, make it possible to find existing pipelines and their associated tools. However, it is still notoriously challenging to develop analysis pipelines that are fully reproducible and interoperable across multiple systems and institutions — primarily because of differences in hardware, operating systems and software versions. Although the recommended guidelines for some analysis pipelines have become standardized (for example, GATK best practices7), the actual implementations are usually developed on a case-by-case basis. As such, there is often little incentive to test, document and implement pipelines in a way that permits their reuse by other researchers. This can hamper sustainable sharing of data and tools, and results in a proliferation of heterogeneous analysis pipelines, making it difficult for newcomers to find what they need to address a specific analysis question. As the scale of -omics data and their associated analytical tools has grown, the scientific community is increasingly moving toward the use of specialized workflow management systems to build analysis pipelines8. They separate the requirements of the underlying compute infrastructure from the analysis and workflow description, introducing a higher degree of portability as compared to custom in-house scripts. One such popular tool is Nextflow9. Using Nextflow, software packages can be bundled with analysis pipelines using built-in integration for package managers, such as Conda, and containerization platforms, such as Docker and Singularity. Moreover, support for most common highperformance-computing batch schedulers and cloud providers allows simple deployment of analysis pipelines on almost any infrastructure. The opportunity to run pipelines locally during initial development and then to proceed seamlessly to largescale computational resources in highperformance-computing or cloud settings provides users and developers with great flexibility. The nf-core community project collects a curated set of best-practice analysis pipelines built using Nextflow. Similar projects Participate
663 citations
TL;DR: Both the updated CONSORT extension for NPT trials and the Consolidated Standards of Reporting Trials extension for abstracts should help authors, editors, and peer reviewers improve the transparency of NPT trial reports.
Abstract: Incomplete and inadequate reporting is an avoidable waste that reduces the usefulness of research. The CONSORT (Consolidated Standards of Reporting Trials) Statement is an evidence-based reporting guideline that aims to improve research transparency and reduce waste. In 2008, the CONSORT Group developed an extension to the original statement that addressed methodological issues specific to trials of nonpharmacologic treatments (NPTs), such as surgery, rehabilitation, or psychotherapy. This article describes an update of that extension and presents an extension for reporting abstracts of NPT trials. To develop these materials, the authors reviewed pertinent literature published up to July 2016; surveyed authors of NPT trials; and conducted a consensus meeting with editors, trialists, and methodologists. Changes to the CONSORT Statement extension for NPT trials include wording modifications to improve readers' understanding and the addition of 3 new items. These items address whether and how adherence of participants to interventions is assessed or enhanced, description of attempts to limit bias if blinding is not possible, and specification of the delay between randomization and initiation of the intervention. The CONSORT extension for abstracts of NPT trials includes 2 new items that were not specified in the original CONSORT Statement for abstracts. The first addresses reporting of eligibility criteria for centers where the intervention is performed and for care providers. The second addresses reporting of important changes to the intervention versus what was planned. Both the updated CONSORT extension for NPT trials and the CONSORT extension for NPT trial abstracts should help authors, editors, and peer reviewers improve the transparency of NPT trial reports.
650 citations
Cites background from "1,500 scientists lift the lid on re..."
...Improving transparency is particularly important in the context of the replication crisis in science (37)....
[...]
References
More filters
Alexander A. Aarts, Joanna E. Anderson1, Christopher J. Anderson2, Peter Raymond Attridge3 +287 more•Institutions (116)
TL;DR: A large-scale assessment suggests that experimental reproducibility in psychology leaves a lot to be desired, and correlational tests suggest that replication success was better predicted by the strength of original evidence than by characteristics of the original and replication teams.
Abstract: Reproducibility is a defining feature of science, but the extent to which it characterizes current research is unknown. We conducted replications of 100 experimental and correlational studies published in three psychology journals using high-powered designs and original materials when available. Replication effects were half the magnitude of original effects, representing a substantial decline. Ninety-seven percent of original studies had statistically significant results. Thirty-six percent of replications had statistically significant results; 47% of original effect sizes were in the 95% confidence interval of the replication effect size; 39% of effects were subjectively rated to have replicated the original result; and if no bias in original results is assumed, combining original and replication results left 68% with statistically significant effects. Correlational tests suggest that replication success was better predicted by the strength of original evidence than by characteristics of the original and replication teams.
5,532 citations
TL;DR: C. Glenn Begley and Lee M. Ellis propose how methods, publications and incentives must change if patients are to benefit.
Abstract: C. Glenn Begley and Lee M. Ellis propose how methods, publications and incentives must change if patients are to benefit.
2,391 citations
TL;DR: The ability to differentiate BMECs from iPSCs grown in mTeSR1 or E-8 medium is supported and it is suggested that the origin of GFR-Matrigel has a marked inpact on BMEC barrier properties.
Abstract: Patient-derived induced pluripotent stem cells (iPSCs) are an innovative source as an in vitro model for neurological diseases. Recent studies have demonstrated the differentiation of brain microvascular endothelial cells (BMECs) from various stem cell sources, including iPSC lines. However, the impact of the culturing conditions used to maintain such stem cell pluripotency on their ability to differentiate into BMECs remains undocumented. In this study, we investigated the effect of different sources of Matrigel and stem cell maintenance medium on BMEC differentiation efficiency. The IMR90-c4 iPSC line was maintained on mTeSR1 or in essential-8 (E-8) medium on growth factor-reduced (GFR) Matrigel from three different manufacturers. Cells were differentiated into BMECs following published protocols. The phenotype of BMEC monolayers was assessed by immunocytochemistry. Barrier function was assessed by transendothelial electrical resistance (TEER) and permeability to sodium fluorescein, whereas the presence of drug efflux pumps was assessed by uptake assay using fluorescent substrates. Stem cell maintenance medium had little effect on the yield and barrier phenotype of IMR90-derived BMECs. The source of GFR-Matrigel used for the differentiation process significantly impacted the ability of IMR90-derived BMECs to form tight monolayers, as measured by TEER and fluorescein permeability. However, the Matrigel source had minimal effect on BMEC phenotype and drug efflux pump activity. This study supports the ability to differentiate BMECs from iPSCs grown in mTeSR1 or E-8 medium and also suggests that the origin of GFR-Matrigel has a marked inpact on BMEC barrier properties.
38 citations
TL;DR: The data demonstrate that although it is very promising against African trypanosomes, cynaropicrin does not show efficacy compared to benznidazole in acute mouse models of T. cruzi infection.
Abstract: In vitro and in vivo activities against Trypanosoma cruzi were evaluated for two sesquiterpene lactones: psilostachyin A and cynaropicrin. Cynaropicrin had previously been shown to potently inhibit African trypanosomes in vivo, and psilostachyin A had been reported to show in vivo effects against T. cruzi, albeit in another test design. In vitro data showed that cynaropicrin was more effective than psilostachyin A. Ultrastructural alterations induced by cynaropicrin included shedding events, detachment of large portions of the plasma membrane, and vesicular bodies and large vacuoles containing membranous structures, suggestive of parasite autophagy. Acute toxicity studies showed that one of two mice died at a cynaropicrin dose of 400 mg/kg of body weight given intraperitoneally (i.p.). Although no major plasma biochemical alterations could be detected, histopathology demonstrated that the liver was the most affected organ in cynaropicrin-treated animals. Although cynaropicrin was as effective as benznidazole against trypomastigotes in vitro, the treatment (once or twice a day) of T. cruzi-infected mice (up to 50 mg/kg/day cynaropicrin) did not suppress parasitemia or protect against mortality induced by the Y and Colombiana strains. Psilostachyin A (0.5 to 50 mg/kg/day given once a day) was not effective in the acute model of T. cruzi infection (Y strain), reaching 100% animal mortality. Our data demonstrate that although it is very promising against African trypanosomes, cynaropicrin does not show efficacy compared to benznidazole in acute mouse models of T. cruzi infection.
33 citations