Cochrane Reviews have recently started including the quantity I 2 to help readers assess the consistency of the results of studies in meta-analyses. What does this new quantity mean, and why is assessment of heterogeneity so important to clinical practice? 

Systematic reviews and meta-analyses can provide convincing and reliable evidence relevant to many aspects of medicine and health care.1 Their value is especially clear when the results of the studies they include show clinically important effects of similar magnitude. However, the conclusions are less clear when the included studies have differing results. In an attempt to establish whether studies are consistent, reports of meta-analyses commonly present a statistical test of heterogeneity. The test seeks to determine whether there are genuine differences underlying the results of the studies (heterogeneity), or whether the variation in findings is compatible with chance alone (homogeneity). However, the test is susceptible to the number of trials included in the meta-analysis. We have developed a new quantity, I 2, which we believe gives a better measure of the consistency between trials in a meta-analysis.

Assessment of the consistency of effects across studies is an essential part of meta-analysis. Unless we know how consistent the results of studies are, we cannot determine the generalisability of the findings of the meta-analysis. Indeed, several hierarchical systems for grading evidence state that the results of studies must be consistent or homogeneous to obtain the highest grading.2–4

Tests for heterogeneity are commonly used to decide on methods for combining studies and for concluding consistency or inconsistency of findings.5 6 But what does the test achieve in practice, and how should the resulting P values be interpreted?

A test for heterogeneity examines the null hypothesis that all studies are evaluating the same effect. The usual test statistic …

/pdf/measuring-inconsistency-in-meta-analyses-4iysmttp08.pdf

Measuring inconsistency in meta-analyses

This report updates the 2002 recommendations by the Advisory Committee on Immunization Practices (ACIP) on the use of influenza vaccine and antiviral agents (CDC. Prevention and Control of Influenza: Recommendations of the Advisory Committee on Immunization Practices [ACIP]. MMWR 2002;51 [No. RR-3]:1-31). The 2003 recommendations include new or updated information regarding 1) the timing of influenza vaccination by age and risk group; 2) influenza vaccine for children aged 6-23 months; 3) the 2003-2004 trivalent inactivated vaccine virus strains: A/Moscow/10/99 (H3N2)-like, A/New Caledonia/20/99 (H1N1)-like, and B/Hong Kong/330/2001-like antigens (for the A/Moscow/10/99 [H3N2]-like antigen, manufacturers will use the antigenically equivalent A/Panama/2007/99 [H3N2] virus, and for the B/Hong Kong/330/2001-like antigen, manufacturers will use either B/Hong Kong/330/2001 or the antigenically equivalent B/Hong Kong/1434/2002); 4) availability of certain influenza vaccine doses with reduced thimerosal content, including single 0.25 mL-dose syringes; and 5) manufacturers of influenza vaccine for the U.S. market. Although the optimal time to vaccinate against influenza is October and November, vaccination in December and later continues to be strongly recommended A link to this report and other information regarding influenza can be accessed at http://www.cdc.gov/ncidod/diseases/flu/fluvirus.htm.

https://stacks.cdc.gov/view/cdc/6625/cdc_6625_DS1.pdf

Prevention and control of influenza : recommendations of the Advisory Committee on Immunization Practices (ACIP)

Guidelines for the management of adult lower respiratory tract infections (LRTIs) were first published by a Task Force of the European Respiratory Society (ERS) in 1998 [1]. In 2005, a completely revised version was produced, this time by a joint Task Force of the ERS and the European Society for Clinical Microbiology and Infectious Diseases (ESCMID) [2]. They used up-to-date methodology for guideline development, including a transparent and objective systematic literature search strategy, and evidence and recommendation grading [2]. Now, 6 yrs later, a joint Task Force of the two Societies, using the same methodology as in 2005, has produced a further update of these guidelines incorporating publications through to May 2010. The Task Force included an epidemiologist with expertise in guideline methodology and experts in the specialty areas relevant to LRTI management, including general practice, microbiology, infectious diseases, respiratory medicine, intensive care and public health.

A short version of the guidelines containing only the recommendations has now been published in Clinical Microbiology and Infection [3], with more detailed versions available on each Society’s website. The guidelines cover the breadth of adult community-acquired respiratory infection, including prevention (both vaccine- and nonvaccine-related), infections in the community and infections in those admitted to hospital, including pneumonia, exacerbations of chronic obstructive pulmonary disease (COPD) and exacerbations of bronchiectasis. The …

https://erj.ersjournals.com/content/erj/26/6/1138.full.pdf

Guidelines for the management of adult lower respiratory tract infections

Objective: Heterogeneity between study results can be a problem in any systematic review or meta-analysis of clinical trials. Identifying its presence, investigating its cause and correctly accounting for it in analyses all involve difficult decisions for the researcher. Our objectives were: to collate recommendations on the subject of dealing with heterogeneity in systematic reviews of clinical trials; to investigate current practice in addressing heterogeneity in Cochrane reviews; and to compare current practice with recommendations.Methods: We review guidelines for those undertaking systematic reviews and examine how heterogeneity is addressed in practice in a sample of systematic reviews, and their protocols, from the Cochrane Database of Systematic Reviews.Results: Advice to reviewers is on the whole consistent and sensible. However, examination of a sample of Cochrane protocols and reviews demonstrates that the advice is difficult to follow given the small numbers of studies identified in many syste...

Statistical heterogeneity in systematic reviews of clinical trials: a critical appraisal of guidelines and practice.

ContextSpecific features of trial quality may be associated with exaggeration
or shrinking of the observed treatment effect in randomized studies. Therefore,
assessment of trial quality is often used in meta-analysis. However, the degree
to which specific quality measures are associated with treatment effects has
not been well established across a broad range of clinical areas.ObjectiveTo determine if quality measures are associated with treatment effect
size in randomized controlled trials (RCTs).DesignQuality measures from published quality assessment scales were evaluated
in RCTs included in meta-analyses from 4 medical areas (cardiovascular disease,
infectious disease, pediatrics, and surgery). Included meta-analyses incorporated
at least 6 RCTs, examined dichotomous outcomes, and demonstrated significant
between-study heterogeneity in the odds ratio (OR) scale.Main Outcome MeasuresRelative ORs comparing overall treatment effect (summary OR) of high
vs low-quality studies, as determined by each quality measure, with relative
ORs less than 1 indicating larger treatment effect in low-quality studies.ResultsTwenty-four quality measures were analyzed for 276 RCTs from 26 meta-analyses.
Relative ORs of high vs low-quality studies for these quality measures ranged
from 0.83 to 1.26; none was statistically significantly associated with treatment
effect. The proportion of studies fulfilling specific quality measures varied
widely in the 4 medical areas. In analyses limited to specific medical areas,
placebo control, multicenter studies, study country, caregiver blinding, and
statistical methods were significantly associated with treatment effect on
7 occasions. These relative ORs ranged from 0.40 to 1.74. However, the directions
of these associations were not consistent.ConclusionsIndividual quality measures are not reliably associated with the strength
of treatment effect across studies and medical areas. Although use of specific
quality measures may be appropriate in specific well-defined areas in which
there is pertinent evidence, findings of associations with treatment effect
cannot be generalized to all clinical areas or meta-analyses.

https://jamanetwork.com/journals/jama/articlepdf/195009/JOC11336.pdf

Correlation of quality measures with estimates of treatment effect in meta-analyses of randomized controlled trials.

BACKGROUND Amantadine hydrochloride and rimantadine hydrochloride have antiviral properties, but these drugs are not widely used due to a lack of knowledge of their potential value and concerns about possible adverse effects. OBJECTIVES The objective of this review was to assess the effectiveness and safety ("effects") of amantadine and rimantadine in healthy adults. SEARCH STRATEGY We searched the Cochrane Central Register of Controlled Trials (CENTRAL) (The Cochrane Library Issue 4, 2003), MEDLINE (January 1966 to November week 2, 2003), EMBASE (January 1990 to September 2003) and the reference lists of articles. We also contacted manufacturers, researchers and authors. SELECTION CRITERIA Randomised and quasi-randomised studies comparing amantadine and/or rimantadine with placebo, control antivirals or no intervention, or comparing doses or schedules of amantadine and/or rimantadine in healthy adults. DATA COLLECTION AND ANALYSIS For prevention trials the numbers of participants with clinical influenza (influenza-like-illness or ILI), i.e. confirmed influenza A, and adverse effects were analysed. Analysis for treatment trials included the mean duration of fever and length of hospital stay, and the number of adverse effects. MAIN RESULTS Amantadine prevented 25% of ILI cases (95% confidence interval (CI) 13% to 36%), and 61% of influenza A cases (95% CI 35% to 76%). Amantadine reduced duration of fever by one day (95% CI 0.7 to 1.3). Rimantadine demonstrated comparable effectiveness, but there were fewer trials and the results for prevention were not statistically significant. Both amantadine and rimantadine induced significant gastrointestinal adverse effects. Adverse effects of the central nervous system and study withdrawals were significantly more common with amantadine than rimantadine. REVIEWERS' CONCLUSIONS Amantadine and rimantadine have comparable effectiveness in the prevention and treatment of influenza A in healthy adults, although rimantadine causes fewer adverse effects than amantadine.

Amantadine and rimantadine for preventing and treating influenza A in adults.

Background Hepatitis B causes acute and chronic liver disease and may be prevented by vaccination. Objectives To assess the effectiveness and safety of plasma-derived vaccines against acute and chronic hepatitis B in health-care workers in protecting them from hepatitis B infection and its consequences. Search strategy MEDLINE and Excerpta Medica Database (EMBASE) search using standard Cochrane strategy, Cochrane Library, full text searching of the journal "Vaccine", bibliography of retrieved studies and correspondence with authors, researchers and manufacturers. Selection criteria All original prospective randomised comparisons of yeast-derived vaccines and plasma-derived vaccines against no intervention, placebo, or vaccines against other disease (control vaccines). Assessment of trial quality was made according to: 1. generation of allocation schedule 2. measure(s) taken to conceal treatment allocation 3. drop-out of allocated health-care worker participants from the analysis of trial results 4. measures taken to implement double blinding Trial reports were blinded by removal of authors and their affiliation, journal reference, introduction, results, and discussion. Data collection and analysis To assess efficacy the incidence rates of acute hepatitis B were observed in the surveillance of the vaccinated and control groups of the trials included in the review. Safety was assessed from side-effect rates, classified as systemic (malaise, nausea, fever, arthralgias, rash, headache) or local (induration and soreness at the site of the inoculation). Main results Four trials fulfilling the criteria were identified and the data synthesised. All trials compared plasma-derived vaccines versus placebo. Differences in the settings (and level of incidence) between three of the trial settings and Dienstag's led us to stratify our comparison grouping the three trials performed in dialysis units together. After our stratification, the Desmyter, Smuzness and Crosnier group appears to be homogeneous (Chi-square = 0.11, degrees of freedom = 2). Our estimates of effectiveness and safety in the high risk group favour treatment, the OR for cases of HB being 0.34; 95% CI (0.21, 0.55). The analysis also revealed a non-significant trend towards benefit in the lower risk health-care workers (Dienstag trial, OR 0.26 (0.05, 1.30). Overall the evidence strongly favours vaccination (OR=0.33; 95% CI (0.21, 0.53)). There was no difference in the incidence and severity of side-effects between the two arms of the trials. We calculated that it was necessary to vaccinate between 145 (assuming a baseline rate of 10 cases/1000/year) and 7 (for a baseline rate of 200/1000/year) health-care workers with plasma-derived vaccines to avoid one case of hepatitis B. Completeness of trial reporting was not good with all four trials failing to report titre results on antibodies against hepatitis B surface antigen and hepatitis B core antigen in the placebo arms (correspondence with two of the four authors failed to shed light on the reasons for such an omission). All four trials achieved low scores in the four quality dimensions assessed (generation of allocation schedule, measure(s) taken to conceal treatment allocation, exclusion of allocated participants from the analysis of the trial and measures taken to implement and protect double blinding). Mean length of follow-up was 14.5 months. Reviewer's conclusions Plasma-derived vaccines appear to be efficacious and safe for use in high risk health-care workers, such as staff of renal dialysis and transplant units. There is some uncertainty concerning the effectiveness of the vaccine in lower risk health-care workers, although the trend is towards benefit. We found no evidence of a long-term protective effect due to the short follow-up time of the four trials included in this review. We found relatively poor standard of trial reporting, possibly related to the age of the trials.

Demicheli

Papers

Amantadine and rimantadine for preventing and treating influenza A in adults.

Vaccines for preventing hepatitis B in health care workers.

Vaccines for preventing hepatitis B in health-care workers.