A key component of scientific communication is sufficient information for other researchers in the field to reproduce published findings. For computational and data-enabled research, this has often been interpreted to mean making available the raw data from which results were generated, the computer code that generated the findings, and any additional information needed such as workflows and input parameters. Many journals are revising author guidelines to include data and code availability. This work evaluates the effectiveness of journal policy that requires the data and code necessary for reproducibility be made available postpublication by the authors upon request. We assess the effectiveness of such a policy by (i) requesting data and code from authors and (ii) attempting replication of the published findings. We chose a random sample of 204 scientific papers published in the journal Science after the implementation of their policy in February 2011. We found that we were able to obtain artifacts from 44% of our sample and were able to reproduce the findings for 26%. We find this policy-author remission of data and code postpublication upon request-an improvement over no policy, but currently insufficient for reproducibility.

https://www.pnas.org/content/pnas/115/11/2584.full.pdf

An empirical analysis of journal policy effectiveness for computational reproducibility.

Preventive interventions are often designed and tested with the immediate program period in mind, and little thought that the intervention sample might be followed up for years or even decades beyond the initial trial. However, depending on the type of intervention and the nature of the outcomes, long-term follow-up may well be appropriate. The advantages of long-term follow-up of preventive interventions are discussed and include the capacity to examine program effects across multiple later life outcomes, the ability to examine the etiological processes involved in the development of the outcomes of interest, and the ability to provide more concrete estimates of the relative benefits and costs of an intervention. In addition, researchers have identified potential methodological risks of long-term follow-up such as inflation of type 1 error through post hoc selection of outcomes, selection bias, and problems stemming from attrition over time. The present paper presents a set of seven recommendations for the design or evaluation of studies for potential long-term follow-up organized under four areas: Intervention Logic Model, Developmental Theory and Measurement Issues; Design for Retention; Dealing with Missing Data; and Unique Considerations for Intervention Studies. These recommendations include conceptual considerations in the design of a study, pragmatic concerns in the design and implementation of the data collection for long-term follow-up, as well as criteria to be considered for the evaluation of an existing intervention for potential for long-term follow-up. Concrete examples from existing intervention studies that have been followed up over the long term are provided.

/pdf/planning-for-long-term-follow-up-strategies-learned-from-4rt4efx0um.pdf

Planning for Long-Term Follow-Up: Strategies Learned from Longitudinal Studies

Health research increasingly relies on organized collections of health data and biological samples. There are many types of sample and data collections that are used for health research, though these are collected for many purposes, not all of which are health-related. These collections exist under different jurisdictional and regulatory arrangements and include:
 Ethical, legal, and social challenges of such collections are well recognized, but there has been limited attention to the broader societal implications of the existence of these collections. Although health research conducted using these collections is broadly recognized as beneficent, secondary uses of these data and samples may be controversial. We examine both documented and hypothetical scenarios of secondary uses of health data and samples. In particular, we focus on the use of health data for purposes of:
 Current safeguards relating to the use of health data and samples include research ethics oversight and privacy laws. These safeguards have a strong focus on informed consent and anonymization, which are aimed at the protection of the individual research subject. They are not intended to address broader societal implications of health data and sample collections. As such, existing arrangements are insufficient to protect against subversion of health databases for non-sanctioned secondary uses, or to provide guidance for reasonable but controversial secondary uses. We are concerned that existing debate in the scholarly literature and beyond has not sufficiently recognized the secondary data uses we outline in this paper. Our main purpose, therefore, is to raise awareness of the potential for unforeseen and unintended consequences, in particular negative consequences, of the increased availability and development of health data collections for research, by providing a comprehensive review of documented and hypothetical non-health research uses of such data.

/pdf/if-you-build-it-they-will-come-unintended-future-uses-of-3gb1k5adgs.pdf

If you build it, they will come: unintended future uses of organised health data collections

Background: Certificates of Confidentiality are intended to facilitate participation in critical public health research by protecting against forced disclosure of identifying data in legal proceedings, but little is known about the effect of Certificate descriptions in consent forms. Methods: To gain preliminary insights, we conducted qualitative interviews with 50 HIV-positive individuals in Durham, NC, to explore their subjective understanding of Certificate descriptions and whether their reactions differed based on receiving a standard versus simplified description. Results: Most interviewees were neither reassured nor alarmed by Certificate information, and most said it would not influence their willingness to participate or to provide truthful information. However, compared with those receiving the simplified description, more who read the standard description said it raised new concerns, that their likelihood of participating would be lower, and that they might be less forthcoming. Most interviewees...

/pdf/research-participants-understanding-of-and-reactions-to-326f5purmt.pdf

Research Participants’ Understanding of and Reactions to Certificates of Confidentiality

Researchers often require and collect sensitive information about individuals to answer important scientific questions that impact individual health and well-being and the public health. Researchers recognize they have a duty to maintain the confidentiality of the data they collect and typically make promises, which are documented in the consent form. The legal interests of others, however, can threaten researchers' promises of confidentiality, if they seek access to the data through subpoena. Certificates of Confidentiality (Certificates), authorized by federal statute, are an important tool for protecting individually identifiable sensitive research data from compelled disclosure. However, questions persist in the research community about the strength of Certificate protections, and the evidence on which to judge the strength is scant. In this article, we address those questions through a careful examination of the legislation and regulations concerning Certificates and the reported and unreported cases we have identified through our legal research and interviews with legal counsel about their experiences with Certificates. We also analyze other statutes that protect research data to compare them to the Certificate's protections, and we review other legal strategies available for protecting research data. Based on our analysis, we conclude with recommendations for how to strengthen protection of sensitive research data.

/pdf/certificates-of-confidentiality-protecting-human-subject-zbwn4jlbhy.pdf

Certificates of Confidentiality: Protecting Human Subject Research Data in Law and Practice

The federal Certificate of Confidentiality plays an important role in research on sensitive topics by authorizing researchers to refuse to disclose identifiable research data in response to subpoenas in any legal setting. However, there is little known about how effective Certificates are in practice. This article draws on our legal and empirical research on this topic to fill this information gap. It includes a description of the purpose of Certificates, their legislative and regulatory history, and a summary of the few reported and unreported cases that have dealt with Certificates. In addition, we outline other statutory confidentiality protections, compare them to the Certificate's protections, and analyze some of the vulnerabilities of a Certificate's protections. This analysis allows us to make specific recommendations for strengthening the protections afforded to research data.

/pdf/certificates-of-confidentiality-protecting-human-subject-jc9eqai7o7.pdf

Certificates of Confidentiality: Protecting Human Subject Research Data in Law and Practice.

The certificate of confidentiality (Certificate) is an important tool for protecting identifiable, sensitive human subjects research data in the United States However, little is known about the Certificate's effectiveness in protecting identifiable data We interviewed 24 legal counsel representing US research institutions about their experiences with legal demands for research data Our respondents reported few, if any, legal demands over the course of their tenure, but two-thirds had experience with legal demands for data protected by a Certificate They reported such demands often were resolved without disclosure of identifiable research data, typically without court intervention While our respondents reported similar success protecting identifiable data in court, they often did not rely on the Certificate to do so

/pdf/certificates-of-confidentiality-legal-counsels-experiences-1ary5epzaw.pdf

Certificates of confidentiality: legal counsels' experiences with and perspectives on legal demands for research data.

Researchers often require and collect sensitive information about individuals to answer important scientific questions that impact individual health and well-being and the public health. Researchers recognize they have a duty to maintain the confidentiality of the data they collect and typically make promises, which are documented in the consent form. The legal interests of others, however, can threaten researchers’ promises of confidentiality, if they seek access to the data through subpoena. Certificates of Confidentiality (Certificates), authorized by federal statute, are an important tool for protecting individually identifiable sensitive research data from compelled disclosure. However, questions persist in the research community about the strength of Certificate protections, and the evidence on which to judge the strength is scant. In this article, we address those questions through a careful examination of the legislation and regulations concerning Certificates and the reported and unreported cases we have identified through our legal research and interviews with legal counsel about their experiences with Certificates. We also analyze other statutes that protect research data to compare them to the Certificate’s protections, and we review other legal strategies available for protecting research data. Based on our analysis, we conclude with recommendations for how to strengthen protection of sensitive research data.

https://papers.ssrn.com/sol3/Delivery.cfm/SSRN_ID2229329_code851442.pdf?abstractid=2217833&mirid=3

Mayank J. Patel

Papers

Certificates of Confidentiality: Protecting Human Subject Research Data in Law and Practice

Certificates of Confidentiality: Protecting Human Subject Research Data in Law and Practice.

Certificates of confidentiality: legal counsels' experiences with and perspectives on legal demands for research data.

Certificates of Confidentiality: Protecting Human Subject Research Data in Law and Practice