Patient and public involvement in the design of clinical trials: An overview of systematic reviews

TL;DR: This systematic overview of PPI included 27 reviews from which areas of good and bad practice were identified and challenges identified include lack of clarity within common language, roles, and research boundaries, while logistical needs include extra time, training, and funding.

Abstract: Background Funders encourage lay-volunteer inclusion in research but this is not without controversy or resistance, given concerns of role confusion, exploratory methods and limited evidence about what value this brings to research. This overview explores these elements. Methods Eleven databases and gray literature were searched without date or language restrictions for systematic reviews of public involvement in clinical trials design. This systematic overview of patient and public involvement (PPI) included 27 reviews from which areas of good and bad practice were identified. PPI strengths, weaknesses, opportunities and threats were explored through use of meta-narrative analysis. Results Inclusion criteria was met by 27 reviews. Confidence in the findings was assessed using Cerqual, Nice-H, CASP for qualitative research and CASP systematic reviews. Quality ranged from high (n=7), medium (n=14) to low (n=6) in the reviews. Four reviews report the risk of bias. Public involvement roles were primarily in agenda setting, steering committees, ethical review, protocol development, and piloting. Research summaries, follow-up, and dissemination contained PPI, with lesser involvement in data collection, analysis, or manuscript authoring. Trialists report difficulty in finding, retaining, and reimbursing volunteers. Respectful inclusion, role recognition, mutual flexibility, advance planning and sound methods were reported as facilitating public involvement in research. Public involvement was reported to have increased the quantity and quality of patient relevant priorities and outcomes, enrollment, funding, design, implementation and dissemination. Challenges identified include lack of clarity within common language, roles and research boundaries; while logistical needs include extra time, training and funding, Researchers report struggling to report involvement and avoid tokenism. Conclusions Involving patients and the public in clinical trials design, can be beneficial but requires resources, preparation, training, flexibility and time. Issues to address include reporting deficits in the areas of risk of bias, study quality and conflicts of interests. There is a need for improved dissemination strategies to increase public involvement and health literacy. Improvements in funding, training, and reporting of PPI are needed to facilitate meaningful and effective PPI.

Summary

Introduction

• The requirements for the planning of Patient (or Personal) and Public Involvement (PPI) in research has increased 1 to encourage research that is ‘with’ or ‘by’ members of the public and patients rather than ‘on,’ ‘to’, ‘about’ or ‘for’ them.
• Many researchers lack PPI 4 training and experience.
• In addition impaired communications between patients, clinicians and researchers are well documented and may obstruct meaningful involvement 5,6 .
• It summarizes what has been found and reported about PPI in clinical trials; identifies the context, methods or processes that facilitate PPI 8 ; collates the perceptions of the influence of PPI on the research process, outcomes.
• And dissemination of results; and promotes the uptake of effective strategies to improve PPI in research and reduce resource costs and that might result from ineffectual PPI.

Why it is important to do this overview

• Research in this field varies in quality, scope, size, and focus, making a systematic overview a practical option 8,9 .
• This enables comparison and critical appraisal of choices made in review selection and can collate, analyze and interpret study results across the separate reviews.

Research Question

• What can the authors learn from existing systematic reviews about involving the public and patients in the design of clinical trials in terms of: 1. How patients and the public are involved in the design of clinical trials 2.
• What is known about good and bad practice for PPI in the design of clinical trials 3.

Objectives

• Identify existing systematic reviews that examine PPI in trials.
• Critically appraise these reviews to assess their methodological quality.
• Extract data from these reviews and use these data to describe how, and to what extent, the public and patients have been involved in trials (other than as participants).
• Seek examples of what worked and what did not to identify good practice.
• Identify research gaps in PPI and trials design.

Research for consideration

• Systematic reviews and overviews published in any language that reviewed existing public or patient involvement in clinical trials (other than as participants) were eligible.
• The involvement could include but was not limited to, prioritization of the research question, involvement in the design or conduct of the trial, analysis, presentation of results, or dissemination of findings.
• Reviews of PPI in clinical trials were eligible if they searched a minimum two databases, appraised the included studies, provided summary findings and included a synthesis of the data and the information retrieved10.

Data Sources and Search Strategy

• The following databases were searched from 1995 until December 2015: Ovid MEDLINE(R) In-Process & Other Non-Indexed Citations and Ovid MEDLINE(R), EMBASE, CINAHL, PsycINFO, Science Citation Index, Cochrane Library, Database of Abstracts of Reviews of Effects, PROSPERO, Global Health Library, Health Technology Assessment, The Joanna Briggs Institute EBP Database, McMaster Knowledge Translation and WHOLIS.
• The PRESS checklist 11 was used to ensure inclusion of essential elements in the search strategy.
• Reference lists and search terms of reviews captured by the initial searches were searched for additional reviews and topic experts were contacted.
• Volunteers were invited to co-create the plain language summary, review the paper for readability and work collaboratively to build an infographic to represent the overview.

Screening and Selection of Reviews

• All citations were screened in RAYYAN 12 , a free online tool that allows the use of unlimited volunteers, tracking and blind review.
• To improve screening accuracy, retrieved citations were screened by one author (AP) and then rescreened by her 4 weeks later.
• Full papers were retrieved for articles that appeared eligible or potentially eligible on the basis of their title and abstract, and for a 1% random sample of those judged to be ineligible to check for correct exclusion 13 .
• Reviewers were not blinded to author, institution, or journal.

Full Paper Retrieval

• Full papers were downloaded to a shared folder and de-duplicated in Mendeley 14 , where overview authors could write and share notes, add questions and additional data.
• Papers were categorized as include, exclude or unsure.
• Papers classified as unsure were discussed and agreement at all stages was reached by consensus of three authors.
• Two review authors independently extracted key data for included reviews, using a data extraction form in EPPI reviewer 15 that was piloted on a small sample of reviews.
• Data on public involvement in clinical trials design and preparation was extracted, covering exploration of roles, policy, impact, reporting, interventions, and theoretical frameworks.

Quality Assessment

• The CASP16 checklists for systematic reviews 17 were used as a preliminary screening tool when assessing systematic reviews for eligibility.
• The first three questions are general and can be used to include or exclude the review.
• The following domains were included when assessing quality: aims, methodology, search quality, recruitment, data collection, data analysis, reflexivity, ethical considerations, findings, and research contributions.

Risk of Bias

• This is based on four components: limitations of methodology, relevance to the research question, coherence and the adequacy of the data presented.
• CerQual enables ratings of “high”, “medium”, “low” and “very low” (although this final rating was not needed because such reviews were not eligible).
• The starting point of ‘high confidence’ reflects that each review finding is a reasonable representation of the question of interest and is downgraded if there are factors that would weaken this assumption 19 .
• After assessing all four components, authors agreed on overall confidence in each review finding and the relevance to their research.
• All measures were pre-specified prior to analysis.

Thematic Analysis

• A strength, weakness, opportunity, and threat (SWOT) framework was used to analyze the findings and organize the data into themes and code them for analysis.
• This made it possible to identify and agree on methods and areas of involvement with positive or negative effect on trial design and to identify research gaps.
• The SWOT approach is used in healthcare research 21,22 to help teams to analyze data individually and then reach consensus on how to present their findings.

Descriptions of Information Presentation Forms

• The excluded reviews summary contains citations and reason for exclusion and is located in results under the heading full-text screening.
• All included reviews contained qualitative elements which meant that the authors need to report their results in a narrative format to describe areas of good and bad practice for PPI in clinical trials and the perceived value of PPI.

Search Report

• Figure 1 uses a Preferred Reporting Items for Systematic Reviews and Meta-Analyses flow diagram to outline the process of study selection 23 .
• Three additional records were found by searching the reference lists of included reviews, one more was included from the EPPI Reviewer database and one review was identified by an expert in the field.

Included Reviews

• Table 1 shows the citation number, first author, year of publication, type of review method, research question focus, number of included studies and funding support type for each included review.
• The total number of studies could not be reported because one review 30 did not report the number of studies reviewed.
• The use and definition of gray literature was variable.
• All academic funding reported was combined with other funding.

Quality Appraisal and Methodological Assessment of Included Reviews

• Conflicts of interest and risk of bias were reported for the included studies32,41,46,49 in four of the included reviews.
• In two reviews this was referred to but not reported by individual study31,49.
• As expected, although some reviews might have been good enough to answer their research question, they were a substandard source of evidence for their research question.
• Likewise, studies seeking impact across research fields 28 contained valuable background information but this was peripheral to the overview aim.

Extent of PPI involvement

• PPI was more frequent in the form of researchers asking members of the public and patients for feedback on the trial design or citizen to citizen interaction such as moderating forums and recruiting participants, rather than in active participation for hands-on research tasks such as study design, ethical review, policy, recruiting, analysis and dissemination.
• PPI impact was reported in 14 reviews using many formats.
• Four reviews 28–30,40 written by authors working together on the question of impact investigated the reporting of impact and have proposed reporting guidelines 61 , however, these reviews were hampered by inconsistent reporting within individual studies.
• Figure 2 shows how PPI was reported across the reviews and the methods of public involvement for various tasks.
• It shows surveys and focus groups were dominant methods of involvement, yet all 27 reviews reported the use of multiple tasks and methods.

Reviewers’ use of PPI in their review

• The authors recorded how review authors reported public involvement in their own reviews to supplement the inconsistent reporting of the numbers of studies or participants involved in tasks.
• This information builds a unique value statement about whether talking about PPI encourages practice.
• Fourteen reviews did not report any PPI in the review and activities were frequently passive.
• The public was updated by review authors and then were invited to advise or comment on the review, rather than engaging the public directly with the data.
• PPI extended to collaborative screening of the literature in three reviews, and analysis and study design roles were largely advisory as recorded in the multiple/other category with seven reviews.

Our PPI for this Systematic Overview

• One volunteer from the Cochrane Task Exchange and three volunteers from Empower-2-Go assisted with screening, data extraction, analysis, synthesis, prioritizing what to report, and editing.
• One lay volunteer and co-author underwent treatment with chemotherapy and radiation for lung cancer and other volunteers completed her tasks.
• Volunteers will help with dissemination planning, conduct and implementation of the overview and are working with us to prepare teaching materials.
• They co-created the plain language summary, suggested improvements for the tables and figures, reviewed the paper for readability and will work collaboratively to build an infographic to represent the overview.

Thematic Analysis with Review Authors and Citizen Collaborators

• The authors report what was learned from existing systematic reviews of primary research for involving the public and patients in the design of clinical trials through a SWOT analysis.
• This allowed us to code the narratives to answer their objectives.

Internal Benefits

• Shared internal benefits of PPI include knowledge of conditions, interventions and expanding of perspectives.
• Negative stereotypes and power imbalances were lessened through working together and were replaced by mutual respect.
• Researchers were encouraged by volunteers’ resiliency, innovation and tenacity and report newly acquired motivation and inspiration to work towards solutions.
• Patients cite greater confidence, research literacy, hope, trust and a sense of community.
• They felt participating gave their lives purpose, meaning, and identity.

External Benefits

• Consultation with volunteers contributed to salient, pragmatic study designs and raised issues that researchers would not otherwise have anticipated.
• Volunteers improved recruiting, interviews, influenced policy setting, and accessed funding for research.
• In addition, there was community influence where PPI was considered a factor in de-stigmatizing mental health, age issues, disease stereotypes, and cultural challenges.
• The external benefits of PPI were reported from early stages in the design of a clinical trial, including in protocol consultation, setting user-focused research objectives and finalizing research questions; developing questionnaires, interview schedules, and consent processes; planning data analysis, user testing, and implementation and dissemination.
• Progress was noted for research awareness, literacy, transparency, and training materials.

Weaknesses of Public Involvement

• Weaknesses were coded by tensions and barriers.
• Shared tensions and barriers were followed by those specific to volunteers, researchers, and organizations.

Tensions in Public Involvement

• Shared tensions revolved around unclear roles, absent or ill-fitting reporting guidelines, tokenism, exclusion, framework limitations, resource allocation and administrative boundaries.
• Jargon was blamed for exclusion and confusion.
• Tensions were balanced by an overarching desire to carve out a mutually agreed path.
• They worried about inappropriate conclusions from composite outcomes but lacked opportunity to share these concerns and noted they would benefit from research methods training.
• Instances were reported where group dynamics changed and overly aggressive patients and those without respect for rules of confidentiality or data protection harmed the research.

Barriers to Public Involvement

• Shared barriers included those imposed by cultures, values, and power hierarchies.
• Patients reported that involving them too late in a trial meant that the design was already funded and fixed and that the priorities and outcomes were not reversible, leaving them with only user experience to contribute.
• They were vulnerable to negative attitudes or dismissive behavior and felt overloaded when drawn into internal strife.
• Researchers struggled to identify a lead for public involvement, a lack of relevant recruiting networks, difficulties with information about structured, practical methods of involvement, and insufficient time to plan for PPI.
• Organizational or gatekeeper barriers ranged from concerns about data being hijacked by opinion rather being centered by evidence.

Recommendations

• A focus on triangulation of teams and clear direction by senior research team members can reduce assumptions.
• Research methods and PPI training increased parity between researchers and volunteers.
• Using flexible responsive approaches to tasks increased the efficiency and quality of involvement.
• Researchers suggested combining interviews and focus groups to reduce scheduling conflicts and manage costs.
• Reviews suggest engaging volunteers in poststudy reflection.

Suggestions

• These suggestions were developed as a result of findings for good and bad practice in PPI and all the included reviews contributed to this theme.
• The table was developed with feedback from volunteers and review authors.
• The tables are used in informal researcher/volunteer training Amy Price developed for The BMJ 68,69 and for Tabula Rasa an asynchronous online medical support learning network 70 .
• The tables were adapted for use in an interactive workshop at The Cochrane Canada Symposium 2017 71 and as an element of the course structure at the FORCE11 Scholarly Communication Institute (FSCI) 2017 72.

Threats to public involvement

• Themes identified as threats are poor reporting, data contamination, ethical breaches, and bad practice.
• All included reviews contributed to this theme.

Poor Reporting

• Threats centered around poor reporting and inadequate quality appraisal of studies and the absence of pre-study published protocols.
• Conflicts of interest revolved around patientprovider relationships, industry and undue influence of advocacy organizations.

Bad Practice

• Planning, training, and information deficits hindered volunteers’ ability to contribute.
• Unpublished methods were lost opportunities for learning.
• Potential harms of PPI need to be balanced against potential benefits with the caveat that patients and carers might be vulnerable populations.
• Supplemental files are valuable for reviewing and learning from the research as they can contain a level of detail that may not be available in the main paper however if they are inaccessible or not linked to the paper their use is limited at best.
• Research students without support are inappropriate for troubleshooting and managing volunteers.

Discussion

• Public involvement was reported as beneficial for volunteers, researchers, and systems in a variety of settings, including different stages of trial design, cultures and disease states.
• As patients become research collaborators, provide recruiting testimonials, conduct interviews with participants and exert cultural change through social media declaring all conflicts of interest would be best practice 75 .
• The overlap across reviews in impact appraisal, research prioritization and choice of outcomes may contribute to an overstatement of equality between researchers and citizens.
• Kirkham and colleagues have defined a methodological approach for assessing the uptake of core outcome sets from findings of randomized controlled trials of rheumatoid arthritis listed on using ClinicalTrials.gov.
• This method may also prove useful for tracking the uptake of PPI and research prioritization 86 .

Limitations of this research

• The absence of dedicated funding for this systematic overview limited double screening to a sample of citations.
• Unspecified MESH terms at the time of the search may have compromised search sensitivity and specificity.
• The deficits in standardized language, research methods and reporting of PPI provided challenges for identifying search terms, assessing quality and risk of bias and this impacted their interpretation of data and scope of comparisons for the overview.
• Differences between protocol and review References in the protocol to quantitative methods, effect sizes, meta-analysis, GRADE 88 , and AMSTAR 89 were not relevant to the final systematic overview because all included reviews were reported qualitatively.
• The authors changed the emphasis to “value reported” rather than “impact reported” because the term impact was based on differing cultural assumptions across disciplines.

Conclusions

• PPI was wide-ranging and innovative in the reviews the authors identified.
• Involving lay volunteers for problem-solving provided insights, enhanced research design and served to identify weaknesses and barriers.
• Contingency plans were useful for adapting to disease progression and competing priorities.
• The use of PPI in dissemination planning, design, implementation, and distribution could increase public involvement, contribute to health literacy and expand knowledge for patient values and preferences.
• Research is evaluated externally by peer review.

Figures

Table 5 Excluded Studies

Table 2 Suggestions for Patient, Personal and Public Involvement

Figure 5 provides an outline of the SWOT with the themes used for analysis

Table 1 Included reviews

Table 4 Overview search terms for MedLine

Figure 3 PPI in reviews (n=27) grouped by task and method, RQ (research question), combination (multiple PPI tasks and methods reported), Multiple/other refers to multiple tasks and other methods such as peer to peer interviewing/support, administering interventions.

Figure 2 Quality Appraisal using CASP, NICE, Risk of bias (ROB)/conflict of interest (COI) and Critical appraisal by CerQual (CQ) appraising four sectors; quality of methodology, coherence, relevance and adequacy and reporting of bias or conflict of interest with scores each ranging from 1-3, low-high for a composite score of 21.

Figure 1 PRISMA Flow Diagram

Table 3 Suggestions for Getting the Best from Public Involvement

Figure 4 Number of included reviews (n=27) and ways the review authors incorporated public involvement

Full text

City, University of London Institutional Repository
Citation: Price, A., Albarqouni, L., Kirkpatrick, J., Clarke, M., Liew, S. M., Roberts, N. and
Burls, A. (2018). Patient and Public Involvement in the Design of Clinical Trials: An Overview
of Systematic Reviews. Journal of Evaluation in Clinical Practice, 24(1), pp. 240-253. doi:
10.1111/jep.12805
This is the accepted version of the paper.
This version of the publication may differ from the final published
version.
Permanent repository link: https://openaccess.city.ac.uk/id/eprint/18382/
Link to published version: http://dx.doi.org/10.1111/jep.12805
Copyright: City Research Online aims to make research outputs of City,
University of London available to a wider audience. Copyright and Moral
Rights remain with the author(s) and/or copyright holders. URLs from
City Research Online may be freely distributed and linked to.
Reuse: Copies of full items can be used for personal research or study,
educational, or not-for-profit purposes without prior permission or
charge. Provided that the authors, title and full bibliographic details are
credited, a hyperlink and/or URL is given for the original metadata page
and the content is not changed in any way.
City Research Online

City Research Online: http://openaccess.city.ac.uk/ publications@city.ac.uk

1
Authors & Affiliations
Amy Price
1
Loai Albarqouni
2
Jo Kirkpatrick
3
Mike Clarke
4
Su May Liew
5
Nia Roberts
1
Amanda Burls
6
University of Oxford
1
Bond University Gold Coast Australia
2
Empower 2 Go
3
Queen’s University Belfast
4
MMed FamMed (Malaya)
5
City, University of London
6
Corresponding Author: Amy Price, Phone 02088842282, email: dr.amyprice@gmail.com

2
Patient and Public Involvement in the
Design of Clinical Trials: An Overview of
Systematic Reviews
Abstract
Background
Funders encourage lay-volunteer inclusion in research but this is not without controversy or
resistance, given concerns of role confusion, exploratory methods and limited evidence about
what value this brings to research. This overview explores these elements.
Methods
Eleven databases and gray literature were searched without date or language restrictions for
systematic reviews of public involvement in clinical trials design. This systematic overview
of patient and public involvement (PPI) included 27 reviews from which areas of good and
bad practice were identified. PPI strengths, weaknesses, opportunities and threats were
explored through use of meta-narrative analysis.
Results
Inclusion criteria was met by 27 reviews. Confidence in the findings was assessed using
Cerqual, Nice-H, CASP for qualitative research and CASP systematic reviews. Quality
ranged from high (n=7), medium (n=14) to low (n=6) in the reviews. Four reviews report the
risk of bias. Public involvement roles were primarily in agenda setting, steering committees,
ethical review, protocol development, and piloting. Research summaries, follow-up, and
dissemination contained PPI, with lesser involvement in data collection, analysis, or
manuscript authoring. Trialists report difficulty in finding, retaining, and reimbursing
volunteers. Respectful inclusion, role recognition, mutual flexibility, advance planning and

3
sound methods were reported as facilitating public involvement in research. Public
involvement was reported to have increased the quantity and quality of patient relevant
priorities and outcomes, enrollment, funding, design, implementation and dissemination.
Challenges identified include lack of clarity within common language, roles and research
boundaries; while logistical needs include extra time, training and funding, Researchers
report struggling to report involvement and avoid tokenism.
Conclusions
Involving patients and the public in clinical trials design, can be beneficial but requires
resources, preparation, training, flexibility and time. Issues to address include reporting
deficits in the areas of risk of bias, study quality and conflicts of interests. There is a need for
improved dissemination strategies to increase public involvement and health literacy.
Improvements in funding, training, and reporting of PPI are needed to facilitate meaningful
and effective PPI.
PROSPERO registration: CRD42016032288 Available from
http://www.crd.york.ac.uk/PROSPERO/display_record.asp?ID=CRD42016032288
PRISMA checklist is available in Appendix-3

Frequently asked questions

Q1. What contributions have the authors mentioned in the paper "Patient and public involvement in the design of clinical trials: an overview of systematic reviews" ?

Background Funders encourage lay-volunteer inclusion in research but this is not without controversy or resistance, given concerns of role confusion, exploratory methods and limited evidence about what value this brings to research. 

Q2. What have the authors stated for future works in "Patient and public involvement in the design of clinical trials: an overview of systematic reviews" ?

Threats to research integrity might be averted through reporting personal conflicts of interest and appraisal of bias in mixed methods or non-quantitative research. The addition of patient reviewers by journals may contribute to health literacy and provide insights for future participatory research practice.