Improving the reliability and efficiency of scientific research will increase the credibility of the published scientific literature and accelerate discovery. Here we argue for the adoption of measures to optimize key elements of the scientific process: methods, reporting and dissemination, reproducibility, evaluation and incentives. There is some evidence from both simulations and empirical studies supporting the likely effectiveness of these measures, but their broad adoption by researchers, institutions, funders and journals will require iterative evaluation and improvement. We discuss the goals of these measures, and how they can be implemented, in the hope that this will facilitate action toward improving the transparency, reproducibility and efficiency of scientific research.

/pdf/a-manifesto-for-reproducible-science-4muga1l76m.pdf

A manifesto for reproducible science

Being able to replicate scientific findings is crucial for scientific progress. We replicate 21 systematically selected experimental studies in the social sciences published in Nature and Science between 2010 and 2015. The replications follow analysis plans reviewed by the original authors and pre-registered prior to the replications. The replications are high powered, with sample sizes on average about five times higher than in the original studies. We find a significant effect in the same direction as the original study for 13 (62%) studies, and the effect size of the replications is on average about 50% of the original effect size. Replicability varies between 12 (57%) and 14 (67%) studies for complementary replicability indicators. Consistent with these results, the estimated true-positive rate is 67% in a Bayesian analysis. The relative effect size of true positives is estimated to be 71%, suggesting that both false positives and inflated effect sizes of true positives contribute to imperfect reproducibility. Furthermore, we find that peer beliefs of replicability are strongly related to replicability, suggesting that the research community could predict which results would replicate and that failures to replicate were not the result of chance alone.

/pdf/evaluating-the-replicability-of-social-science-experiments-5c0dkci5cq.pdf

Evaluating the replicability of social science experiments in Nature and Science between 2010 and 2015

We present Bioconda (https://bioconda.github.io), a distribution of bioinformatics software for the lightweight, multi-platform and language-agnostic package manager Conda. Currently, Bioconda offers a collection of over 3000 software packages, which is continuously maintained, updated, and extended by a growing global community of more than 200 contributors. Bioconda improves analysis reproducibility by allowing users to define isolated environments with defined software versions, all of which are easily installed and managed without administrative privileges.

/pdf/bioconda-sustainable-and-comprehensive-software-distribution-2aas0o7u44.pdf

Bioconda: sustainable and comprehensive software distribution for the life sciences.

To the Editor — The standardization, portability and reproducibility of analysis pipelines are key issues within the bioinformatics community. Most bioinformatics pipelines are designed for use on-premises; as a result, the associated software dependencies and execution logic are likely to be tightly coupled with proprietary computing environments. This can make it difficult or even impossible for others to reproduce the ensuing results, which is a fundamental requirement for the validation of scientific findings. Here, we introduce the nf-core framework as a means for the development of collaborative, peerreviewed, best-practice analysis pipelines (Fig. 1). All nf-core pipelines are written in Nextflow and so inherit the ability to be executed on most computational infrastructures, as well as having native support for container technologies such as Docker and Singularity. The nf-core community (Supplementary Fig. 1) has developed a suite of tools that automate pipeline creation, testing, deployment and synchronization. Our goal is to provide a framework for high-quality bioinformatics pipelines that can be used across all institutions and research facilities. Being able to reproduce scientific results is the central tenet of the scientific method. However, moving toward FAIR (findable, accessible, interoperable and reusable) research methods1 in data-driven science is complex2,3. Central repositories, such as bio. tools4, omictools5 and the Galaxy toolshed6, make it possible to find existing pipelines and their associated tools. However, it is still notoriously challenging to develop analysis pipelines that are fully reproducible and interoperable across multiple systems and institutions — primarily because of differences in hardware, operating systems and software versions. Although the recommended guidelines for some analysis pipelines have become standardized (for example, GATK best practices7), the actual implementations are usually developed on a case-by-case basis. As such, there is often little incentive to test, document and implement pipelines in a way that permits their reuse by other researchers. This can hamper sustainable sharing of data and tools, and results in a proliferation of heterogeneous analysis pipelines, making it difficult for newcomers to find what they need to address a specific analysis question. As the scale of -omics data and their associated analytical tools has grown, the scientific community is increasingly moving toward the use of specialized workflow management systems to build analysis pipelines8. They separate the requirements of the underlying compute infrastructure from the analysis and workflow description, introducing a higher degree of portability as compared to custom in-house scripts. One such popular tool is Nextflow9. Using Nextflow, software packages can be bundled with analysis pipelines using built-in integration for package managers, such as Conda, and containerization platforms, such as Docker and Singularity. Moreover, support for most common highperformance-computing batch schedulers and cloud providers allows simple deployment of analysis pipelines on almost any infrastructure. The opportunity to run pipelines locally during initial development and then to proceed seamlessly to largescale computational resources in highperformance-computing or cloud settings provides users and developers with great flexibility. The nf-core community project collects a curated set of best-practice analysis pipelines built using Nextflow. Similar projects Participate

The nf-core framework for community-curated bioinformatics pipelines.

Incomplete and inadequate reporting is an avoidable waste that reduces the usefulness of research. The CONSORT (Consolidated Standards of Reporting Trials) Statement is an evidence-based reporting guideline that aims to improve research transparency and reduce waste. In 2008, the CONSORT Group developed an extension to the original statement that addressed methodological issues specific to trials of nonpharmacologic treatments (NPTs), such as surgery, rehabilitation, or psychotherapy. This article describes an update of that extension and presents an extension for reporting abstracts of NPT trials. To develop these materials, the authors reviewed pertinent literature published up to July 2016; surveyed authors of NPT trials; and conducted a consensus meeting with editors, trialists, and methodologists. Changes to the CONSORT Statement extension for NPT trials include wording modifications to improve readers' understanding and the addition of 3 new items. These items address whether and how adherence of participants to interventions is assessed or enhanced, description of attempts to limit bias if blinding is not possible, and specification of the delay between randomization and initiation of the intervention. The CONSORT extension for abstracts of NPT trials includes 2 new items that were not specified in the original CONSORT Statement for abstracts. The first addresses reporting of eligibility criteria for centers where the intervention is performed and for care providers. The second addresses reporting of important changes to the intervention versus what was planned. Both the updated CONSORT extension for NPT trials and the CONSORT extension for NPT trial abstracts should help authors, editors, and peer reviewers improve the transparency of NPT trial reports.

/pdf/consort-statement-for-randomized-trials-of-nonpharmacologic-4uqflkox34.pdf

CONSORT Statement for Randomized Trials of Nonpharmacologic Treatments: A 2017 Update and a CONSORT Extension for Nonpharmacologic Trial Abstracts

/pdf/1-500-scientists-lift-the-lid-on-reproducibility-58wbka41cp.pdf

1,500 scientists lift the lid on reproducibility

'Reproducibility editor' Victoria Stodden explains the growing movement to make code and data available to others.

Why scientists must share their research code

Researchers tease out different definitions of a crucial scientific term.

Muddled meanings hamper efforts to fix reproducibility crisis

Three-year pilot devotes €3 million to verifying other studies.

Dutch agency launches first grants programme dedicated to replication

Reanalysis of last year's enormous replication study argues that there is no need to be so pessimistic.

Monya Baker

Papers

1,500 scientists lift the lid on reproducibility

Why scientists must share their research code

Muddled meanings hamper efforts to fix reproducibility crisis

Dutch agency launches first grants programme dedicated to replication

Psychology’s reproducibility problem is exaggerated – say psychologists