Cochrane Reviews have recently started including the quantity I 2 to help readers assess the consistency of the results of studies in meta-analyses. What does this new quantity mean, and why is assessment of heterogeneity so important to clinical practice? 

Systematic reviews and meta-analyses can provide convincing and reliable evidence relevant to many aspects of medicine and health care.1 Their value is especially clear when the results of the studies they include show clinically important effects of similar magnitude. However, the conclusions are less clear when the included studies have differing results. In an attempt to establish whether studies are consistent, reports of meta-analyses commonly present a statistical test of heterogeneity. The test seeks to determine whether there are genuine differences underlying the results of the studies (heterogeneity), or whether the variation in findings is compatible with chance alone (homogeneity). However, the test is susceptible to the number of trials included in the meta-analysis. We have developed a new quantity, I 2, which we believe gives a better measure of the consistency between trials in a meta-analysis.

Assessment of the consistency of effects across studies is an essential part of meta-analysis. Unless we know how consistent the results of studies are, we cannot determine the generalisability of the findings of the meta-analysis. Indeed, several hierarchical systems for grading evidence state that the results of studies must be consistent or homogeneous to obtain the highest grading.2–4

Tests for heterogeneity are commonly used to decide on methods for combining studies and for concluding consistency or inconsistency of findings.5 6 But what does the test achieve in practice, and how should the resulting P values be interpreted?

A test for heterogeneity examines the null hypothesis that all studies are evaluating the same effect. The usual test statistic …

/pdf/measuring-inconsistency-in-meta-analyses-4iysmttp08.pdf

Measuring inconsistency in meta-analyses

Objective: Funnel plots (plots of effect estimates against sample size) may be useful to detect bias in meta-analyses that were later contradicted by large trials. We examined whether a simple test of asymmetry of funnel plots predicts discordance of results when meta-analyses are compared to large trials, and we assessed the prevalence of bias in published meta-analyses. Design: Medline search to identify pairs consisting of a meta-analysis and a single large trial (concordance of results was assumed if effects were in the same direction and the meta-analytic estimate was within 30% of the trial); analysis of funnel plots from 37 meta-analyses identified from a hand search of four leading general medicine journals 1993-6 and 38 meta-analyses from the second 1996 issue of the Cochrane Database of Systematic Reviews . Main outcome measure: Degree of funnel plot asymmetry as measured by the intercept from regression of standard normal deviates against precision. Results: In the eight pairs of meta-analysis and large trial that were identified (five from cardiovascular medicine, one from diabetic medicine, one from geriatric medicine, one from perinatal medicine) there were four concordant and four discordant pairs. In all cases discordance was due to meta-analyses showing larger effects. Funnel plot asymmetry was present in three out of four discordant pairs but in none of concordant pairs. In 14 (38%) journal meta-analyses and 5 (13%) Cochrane reviews, funnel plot asymmetry indicated that there was bias. Conclusions: A simple analysis of funnel plots provides a useful test for the likely presence of bias in meta-analyses, but as the capacity to detect bias will be limited when meta-analyses are based on a limited number of small trials the results from such analyses should be treated with considerable caution. Key messages Systematic reviews of randomised trials are the best strategy for appraising evidence; however, the findings of some meta-analyses were later contradicted by large trials Funnel plots, plots of the trials9 effect estimates against sample size, are skewed and asymmetrical in the presence of publication bias and other biases Funnel plot asymmetry, measured by regression analysis, predicts discordance of results when meta-analyses are compared with single large trials Funnel plot asymmetry was found in 38% of meta-analyses published in leading general medicine journals and in 13% of reviews from the Cochrane Database of Systematic Reviews Critical examination of systematic reviews for publication and related biases should be considered a routine procedure

/pdf/bias-in-meta-analysis-detected-by-a-simple-graphical-test-5erh428s5c.pdf

Bias in meta-analysis detected by a simple, graphical test

Systematic review and meta-analysis

Systematic reviews and meta-analyses are essential to summarize evidence relating to efficacy and safety of health care interventions accurately and reliably. The clarity and transparency of these reports, however, is not optimal. Poor reporting of systematic reviews diminishes their value to clinicians, policy makers, and other users.

Since the development of the QUOROM (QUality Of Reporting Of Meta-analysis) Statement—a reporting guideline published in 1999—there have been several conceptual, methodological, and practical advances regarding the conduct and reporting of systematic reviews and meta-analyses. Also, reviews of published systematic reviews have found that key information about these studies is often poorly reported. Realizing these issues, an international group that included experienced authors and methodologists developed PRISMA (Preferred Reporting Items for Systematic reviews and Meta-Analyses) as an evolution of the original QUOROM guideline for systematic reviews and meta-analyses of evaluations of health care interventions.

The PRISMA Statement consists of a 27-item checklist and a four-phase flow diagram. The checklist includes items deemed essential for transparent reporting of a systematic review. In this Explanation and Elaboration document, we explain the meaning and rationale for each checklist item. For each item, we include an example of good reporting and, where possible, references to relevant empirical studies and methodological literature. The PRISMA Statement, this document, and the associated Web site (http://www.prisma-statement.org/) should be helpful resources to improve reporting of systematic reviews and meta-analyses.

/pdf/the-prisma-statement-for-reporting-systematic-reviews-and-3j53e5oe33.pdf

The PRISMA Statement for Reporting Systematic Reviews and Meta-Analyses of Studies That Evaluate Health Care Interventions: Explanation and Elaboration

The Cochrane Handbook for Systematic Reviews of Interventions is the official document that describes in detail the process of preparing and maintaining Cochrane systematic reviews on the effects of healthcare interventions.

/pdf/cochrane-handbook-for-systematic-reviews-of-interventions-1xbdy6uvru.pdf

Cochrane Handbook for Systematic Reviews of Interventions

Clinical studies in psychiatry frequently involve the analysis of matched pairs, in which each patient is matched with his or her psychiatric clinician. If the response variable of interest is dichotomous, then McNemar's chi 2 test for correlated proportions may be applied to the resulting data. However, this test assumes that the set of matched pairs are statistically independent, an assumption that is violated if some clinicians are matched with more than one patient. In this article we present alternatives to McNemar's test for dealing with this problem.

The statistical analysis of matched data in psychiatric research.

M1079 Improving Adherence to Guidelines When Managing Non-Variceal Upper Gastrointestinal Bleeding: A National Cluster Randomized Trial of a Multifaceted Strategy

The use of individualized regressions, which reduces the polychotomous logistic regression model to several dichotomous models, has been proposed as a solution to some practical difficulties for binary covariates (Begg and Gray 1984, Biometrika, 71, 11–18). Its disadvantages, however, include loss of efficiency and the complexity of making comparisons among regressions. Using expressions for the large-sample distribution of the maximum-likelihood estimates, the efficiency of the individualized procedure relative to the polychotomous procedure is evaluated for the case in which the covariates are assumed to follow a multivariate normal distribution. The relative efficiency when the logistic slope vectors from different regressions are collinear can be substantially lower compared to the efficiency with orthogonal slope vectors. Further evaluations for binary covariates using collinear and orthogonal slope parametrizations lead to a similar characterization.



L'emploi de regressions individualisees, reduisant le modele de regression logistique polychotomique a plusieurs modeles dichotomiques, a ete propose comme solution a quelques difficultes pratiques pour les facteurs binaires (Begg et Gray 1984, Biometrika, 71, 11–18). Cette methode a cependant quelques desavantages, dont une perte d'efficacite et la difficulte d'etablir des comparaisons entre regressions. L'efficacite asymptotique relative de la procedure individualisee par rapport a la procedure polychotomique est evaluee a l'aide de la theorie asymptotique des estimateurs du maximum de vraisemblance, dans le cas ou les facteurs ont une distribution normale multivariee. L'efficacite relative, lorsque les vecteurs de pente logistiques de differentes regressions sont collineaires, peut s'averer considerablement inferieure a l'efficacite lorsque les vecteurs de pente sont orthogonaux. Des evaluations supplementaires pour des facteurs binaires employant des parametrisations de pente collineaires et orthogonales produisent une caracterisation semblable.

A characterization of the efficiency of individualized logistic regressions

Various strategies are investigated for selecting one of two medical treatments when patients may be divided into k ≥ 2 categories on the basis of their expected differences in response to the two treatments. The strategies compared are (1) k independent decisions, (2) a single overall decision made on the basis of simple pooling, and (3) the Bayes strategy. The optimal clinical trial size is supplied for each strategy, and conditions delineated under which each of the strategies is to be preferred.



On etudie diverses statesies pour choisir un traitement dedical parmi deux, lorsque les patients peuvent etre repartis en k ≥ 2 categories selon la defference entre leurs reactions problables aux deux traitements. Les strategies comparees sont: 1 k decisions independantes, 2 une seule decision globale basee sur les resultats combines, et 3 la strategie de Bayes. La taille optimale de l'essai clinique est fournie pour chaque strates, et 3 la strategie, et les conditions sous lesquelles chacune des strategies est la meilleure sont enoncees.

http://www.jstor.org/stable/pdfplus/3315116.pdf

The design of a clinical trial with several patient categories

Rosner (1983, 1984) proposed a model for correlated binary outcomes in the presence of covariates based on the beta-binomial distribution. Although calculation of maximum likelihood estimates under this model is costly, alternative estimators based on the usual logistic model, with or without dummy varibles, are asymptotically biased (in general) and yield standard errors that are incorrect. An adjustment for the standard error of the usual logistic estimator is suggested for the case of one dichotomous independent variable. Also a familiar conditional approach is shown to have low asymptotic relative efficiency.

Allan Donner

Papers

The statistical analysis of matched data in psychiatric research.

M1079 Improving Adherence to Guidelines When Managing Non-Variceal Upper Gastrointestinal Bleeding: A National Cluster Randomized Trial of a Multifaceted Strategy

A characterization of the efficiency of individualized logistic regressions

The design of a clinical trial with several patient categories

Estimation Under the Correlated Logistic Model