David Moher and colleagues introduce PRISMA, an update of the QUOROM guidelines for reporting systematic reviews and meta-analyses

/pdf/preferred-reporting-items-for-systematic-reviews-and-meta-3k0wa2mp7o.pdf

Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement

Systematic reviews and meta-analyses have become increasingly important in health care. Clinicians read them to keep up to date with their field,1,2 and they are often used as a starting point for developing clinical practice guidelines. Granting agencies may require a systematic review to ensure there is justification for further research,3 and some health care journals are moving in this direction.4 As with all research, the value of a systematic review depends on what was done, what was found, and the clarity of reporting. As with other publications, the reporting quality of systematic reviews varies, limiting readers' ability to assess the strengths and weaknesses of those reviews.

Several early studies evaluated the quality of review reports. In 1987, Mulrow examined 50 review articles published in 4 leading medical journals in 1985 and 1986 and found that none met all 8 explicit scientific criteria, such as a quality assessment of included studies.5 In 1987, Sacks and colleagues6 evaluated the adequacy of reporting of 83 meta-analyses on 23 characteristics in 6 domains. Reporting was generally poor; between 1 and 14 characteristics were adequately reported (mean = 7.7; standard deviation = 2.7). A 1996 update of this study found little improvement.7

In 1996, to address the suboptimal reporting of meta-analyses, an international group developed a guidance called the QUOROM Statement (QUality Of Reporting Of Meta-analyses), which focused on the reporting of meta-analyses of randomized controlled trials.8 In this article, we summarize a revision of these guidelines, renamed PRISMA (Preferred Reporting Items for Systematic reviews and Meta-Analyses), which have been updated to address several conceptual and practical advances in the science of systematic reviews (Box 1).



Box 1

Conceptual issues in the evolution from QUOROM to PRISMA

Preferred reporting items for systematic reviews and meta-analyses: the PRISMA Statement.

Objective: Funnel plots (plots of effect estimates against sample size) may be useful to detect bias in meta-analyses that were later contradicted by large trials. We examined whether a simple test of asymmetry of funnel plots predicts discordance of results when meta-analyses are compared to large trials, and we assessed the prevalence of bias in published meta-analyses. Design: Medline search to identify pairs consisting of a meta-analysis and a single large trial (concordance of results was assumed if effects were in the same direction and the meta-analytic estimate was within 30% of the trial); analysis of funnel plots from 37 meta-analyses identified from a hand search of four leading general medicine journals 1993-6 and 38 meta-analyses from the second 1996 issue of the Cochrane Database of Systematic Reviews . Main outcome measure: Degree of funnel plot asymmetry as measured by the intercept from regression of standard normal deviates against precision. Results: In the eight pairs of meta-analysis and large trial that were identified (five from cardiovascular medicine, one from diabetic medicine, one from geriatric medicine, one from perinatal medicine) there were four concordant and four discordant pairs. In all cases discordance was due to meta-analyses showing larger effects. Funnel plot asymmetry was present in three out of four discordant pairs but in none of concordant pairs. In 14 (38%) journal meta-analyses and 5 (13%) Cochrane reviews, funnel plot asymmetry indicated that there was bias. Conclusions: A simple analysis of funnel plots provides a useful test for the likely presence of bias in meta-analyses, but as the capacity to detect bias will be limited when meta-analyses are based on a limited number of small trials the results from such analyses should be treated with considerable caution. Key messages Systematic reviews of randomised trials are the best strategy for appraising evidence; however, the findings of some meta-analyses were later contradicted by large trials Funnel plots, plots of the trials9 effect estimates against sample size, are skewed and asymmetrical in the presence of publication bias and other biases Funnel plot asymmetry, measured by regression analysis, predicts discordance of results when meta-analyses are compared with single large trials Funnel plot asymmetry was found in 38% of meta-analyses published in leading general medicine journals and in 13% of reviews from the Cochrane Database of Systematic Reviews Critical examination of systematic reviews for publication and related biases should be considered a routine procedure

/pdf/bias-in-meta-analysis-detected-by-a-simple-graphical-test-5erh428s5c.pdf

Bias in meta-analysis detected by a simple, graphical test

Systematic review and meta-analysis

Preferred reporting items for systematic reviews and meta-analyses: The PRISMA statement

Background
Placebo interventions are often believed to improve patient reported and observer reported outcomes, but this belief is not based on evidence from randomised trials that compare placebo with no treatment.

Objectives
To assess the effect of placebo interventions.

Search strategy
We searched the Cochrane Controlled Trials Register (The Cochrane Library, issue 3, 1998), MEDLINE (Jan 1966 to Dec 1998), EMBASE (Jan 1980 to Dec 1998), Biological Abstracts (Jan 1986 to Dec 1998), PsycLIT (Jan 1887 to Dec 1998). Experts on placebo research were contacted and references in the included trials were read.

Selection criteria
Randomised placebo trials with a no-treatment control group investigating any health problem were included.

Data collection and analysis
Two reviewers independently assessed trial quality and extracted data. Study authors were contacted for additional information.

Main results
Outcome data were available in 114 out of 130 included trials, investigating 40 clinical conditions. Outcomes were binary in 32 trials (3795 patients) and continuous in 82 (4730 patients).

We found no statistically significant pooled effect of placebo in studies with binary outcomes, relative risk 0.95 (95 per cent confidence interval 0.88 to 1.02). The pooled relative risk for subjective (patient reported) outcomes was 0.95 (0.86 to 1.05) and for objective (observer reported) outcomes 0.91 (0.80 to 1.04). There was statistically significant heterogeneity (P < 0.03), but no evidence of sample size bias (P = 0.56).

We found an overall positive effect of placebo treatments in trials with continuous outcomes, standardised mean difference -0.28 (95 per cent confidence interval -0.38 to -0.19). The standardised mean difference for subjective outcomes was -0.36 (-0.47 to -0.25), whereas no statistically significant effect was found for objective outcomes, standardised mean difference -0.12 (-0.27 to 0.03). There was statistically significant heterogeneity (P < 0.001), and evidence of sample size bias (P = 0.05).

There was no statistically significant effect of placebo interventions in eight out of nine clinical conditions investigated in three trials or more (nausea, relapse in prevention of smoking and depression, overweight, asthma, hypertension, insomnia and anxiety), but confidence intervals were wide. There was a modest apparent analgesic effect of placebo interventions, standardised mean difference -0.27 (-0.40 to -0.15), but also a substantial risk of bias.

Reviewers' conclusions
There was no evidence that placebo interventions in general have clinically important effects. A possible moderate effect on subjective continuous outcomes, especially pain, could not be clearly distinguished from bias.

Placebo treatment versus no treatment

Unsubstantiated claims of large effects of placebo on pain: serious errors in meta-analysis of placebo analgesia mechanism studies

A study found that women participating in mammography screening were content with the programme and the paternalistic invitations that directly encourage participation and include a pre-specified time of appointment. We argue that this merely reflects that the information presented to the invited women is seriously biased in favour of participation. Women are not informed about the major harms of screening, and the decision to attend has already been made for them by a public authority. This short-circuits informed decision-making and the legislation on informed consent, and violates the autonomy of the women. Screening invitations must present both benefits and harms in a balanced fashion, and should offer, not encourage, participation. It should be stated clearly that the choice not to participate is as sensible as the choice to do so. To allow this to happen, the responsibility for the screening programmes must be separated from the responsibility for the information material.

https://jme.bmj.com/content/medethics/35/4/268.full.pdf

Informed choice requires information about both benefits and harms

In May 2018, the Cochrane Collaboration published its review of the human papillomavirus (HPV) vaccines.1 The review primarily assessed the vaccines’ effect on precursors to cervical cancer. Cochrane has high standards for its reviews2; however, there were important limitations in its HPV vaccine review, which we address in this paper.

The Cochrane review conducted trial searches up until June 2017 and included 26 randomised trials with 73 428 women.1 In January 2018, we published an index of the study programmes of the HPV vaccines that included 206 comparative studies.3 As of June 2017, about one-third of the 206 studies were not published and half of the completed studies listed on ClinicalTrials.gov had no results posted.3 Although we sent our index to the Cochrane group handling the Cochrane review, the review stated that, ‘nearly all end-of-study reports have been published in the peer-reviewed literature’. When we applied the Cochrane review’s inclusion criteria to the 206 studies, we identified 46 completed and eligible trials. The number of randomised participants could be assessed for 42 of the 46 trials and was 121 704. With nearly half of the trials and half of the participants missing, the Cochrane authors’ conclusion, ‘that the risk of reporting bias may be small’, was inappropriate. Fifteen of the 20 additional trials were listed on ClinicalTrials.gov; the Cochrane authors would therefore have identified more trials if they had searched ClinicalTrials.gov in more depth and searched additional trial registers (we searched 45 trial registers3).

The Cochrane authors stated that they ‘did not include …

https://ebm.bmj.com/content/ebmed/23/5/165.full.pdf

The Cochrane HPV vaccine review was incomplete and ignored important evidence of bias.

Objectives To examine the extent of multiplicity of data in trial reports and to assess the impact of multiplicity on meta-analysis results.

Design Empirical study on a cohort of Cochrane systematic reviews.

Data sources All Cochrane systematic reviews published from issue 3 in 2006 to issue 2 in 2007 that presented a result as a standardised mean difference (SMD). We retrieved trial reports contributing to the first SMD result in each review, and downloaded review protocols. We used these SMDs to identify a specific outcome for each meta-analysis from its protocol.

Review methods Reviews were eligible if SMD results were based on two to ten randomised trials and if protocols described the outcome. We excluded reviews if they only presented results of subgroup analyses. Based on review protocols and index outcomes, two observers independently extracted the data necessary to calculate SMDs from the original trial reports for any intervention group, time point, or outcome measure compatible with the protocol. From the extracted data, we used Monte Carlo simulations to calculate all possible SMDs for every meta-analysis.

Results We identified 19 eligible meta-analyses (including 83 trials). Published review protocols often lacked information about which data to choose. Twenty-four (29%) trials reported data for multiple intervention groups, 30 (36%) reported data for multiple time points, and 29 (35%) reported the index outcome measured on multiple scales. In 18 meta-analyses, we found multiplicity of data in at least one trial report; the median difference between the smallest and largest SMD results within a meta-analysis was 0.40 standard deviation units (range 0.04 to 0.91).

Conclusions Multiplicity of data can affect the findings of systematic reviews and meta-analyses. To reduce the risk of bias, reviews and meta-analyses should comply with prespecified protocols that clearly identify time points, intervention groups, and scales of interest.

https://www.bmj.com/content/bmj/343/bmj.d4829.full.pdf

Peter C Gøtzsche

Papers

Placebo treatment versus no treatment

Unsubstantiated claims of large effects of placebo on pain: serious errors in meta-analysis of placebo analgesia mechanism studies

Informed choice requires information about both benefits and harms

The Cochrane HPV vaccine review was incomplete and ignored important evidence of bias.

Multiplicity of data in trial reports and the reliability of meta-analyses: empirical study.