Cochrane Reviews have recently started including the quantity I 2 to help readers assess the consistency of the results of studies in meta-analyses. What does this new quantity mean, and why is assessment of heterogeneity so important to clinical practice? 

Systematic reviews and meta-analyses can provide convincing and reliable evidence relevant to many aspects of medicine and health care.1 Their value is especially clear when the results of the studies they include show clinically important effects of similar magnitude. However, the conclusions are less clear when the included studies have differing results. In an attempt to establish whether studies are consistent, reports of meta-analyses commonly present a statistical test of heterogeneity. The test seeks to determine whether there are genuine differences underlying the results of the studies (heterogeneity), or whether the variation in findings is compatible with chance alone (homogeneity). However, the test is susceptible to the number of trials included in the meta-analysis. We have developed a new quantity, I 2, which we believe gives a better measure of the consistency between trials in a meta-analysis.

Assessment of the consistency of effects across studies is an essential part of meta-analysis. Unless we know how consistent the results of studies are, we cannot determine the generalisability of the findings of the meta-analysis. Indeed, several hierarchical systems for grading evidence state that the results of studies must be consistent or homogeneous to obtain the highest grading.2–4

Tests for heterogeneity are commonly used to decide on methods for combining studies and for concluding consistency or inconsistency of findings.5 6 But what does the test achieve in practice, and how should the resulting P values be interpreted?

A test for heterogeneity examines the null hypothesis that all studies are evaluating the same effect. The usual test statistic …

/pdf/measuring-inconsistency-in-meta-analyses-4iysmttp08.pdf

Measuring inconsistency in meta-analyses

Objective: Funnel plots (plots of effect estimates against sample size) may be useful to detect bias in meta-analyses that were later contradicted by large trials. We examined whether a simple test of asymmetry of funnel plots predicts discordance of results when meta-analyses are compared to large trials, and we assessed the prevalence of bias in published meta-analyses. Design: Medline search to identify pairs consisting of a meta-analysis and a single large trial (concordance of results was assumed if effects were in the same direction and the meta-analytic estimate was within 30% of the trial); analysis of funnel plots from 37 meta-analyses identified from a hand search of four leading general medicine journals 1993-6 and 38 meta-analyses from the second 1996 issue of the Cochrane Database of Systematic Reviews . Main outcome measure: Degree of funnel plot asymmetry as measured by the intercept from regression of standard normal deviates against precision. Results: In the eight pairs of meta-analysis and large trial that were identified (five from cardiovascular medicine, one from diabetic medicine, one from geriatric medicine, one from perinatal medicine) there were four concordant and four discordant pairs. In all cases discordance was due to meta-analyses showing larger effects. Funnel plot asymmetry was present in three out of four discordant pairs but in none of concordant pairs. In 14 (38%) journal meta-analyses and 5 (13%) Cochrane reviews, funnel plot asymmetry indicated that there was bias. Conclusions: A simple analysis of funnel plots provides a useful test for the likely presence of bias in meta-analyses, but as the capacity to detect bias will be limited when meta-analyses are based on a limited number of small trials the results from such analyses should be treated with considerable caution. Key messages Systematic reviews of randomised trials are the best strategy for appraising evidence; however, the findings of some meta-analyses were later contradicted by large trials Funnel plots, plots of the trials9 effect estimates against sample size, are skewed and asymmetrical in the presence of publication bias and other biases Funnel plot asymmetry, measured by regression analysis, predicts discordance of results when meta-analyses are compared with single large trials Funnel plot asymmetry was found in 38% of meta-analyses published in leading general medicine journals and in 13% of reviews from the Cochrane Database of Systematic Reviews Critical examination of systematic reviews for publication and related biases should be considered a routine procedure

/pdf/bias-in-meta-analysis-detected-by-a-simple-graphical-test-5erh428s5c.pdf

Bias in meta-analysis detected by a simple, graphical test

Systematic review and meta-analysis

Systematic reviews and meta-analyses are essential to summarize evidence relating to efficacy and safety of health care interventions accurately and reliably. The clarity and transparency of these reports, however, is not optimal. Poor reporting of systematic reviews diminishes their value to clinicians, policy makers, and other users.

Since the development of the QUOROM (QUality Of Reporting Of Meta-analysis) Statement—a reporting guideline published in 1999—there have been several conceptual, methodological, and practical advances regarding the conduct and reporting of systematic reviews and meta-analyses. Also, reviews of published systematic reviews have found that key information about these studies is often poorly reported. Realizing these issues, an international group that included experienced authors and methodologists developed PRISMA (Preferred Reporting Items for Systematic reviews and Meta-Analyses) as an evolution of the original QUOROM guideline for systematic reviews and meta-analyses of evaluations of health care interventions.

The PRISMA Statement consists of a 27-item checklist and a four-phase flow diagram. The checklist includes items deemed essential for transparent reporting of a systematic review. In this Explanation and Elaboration document, we explain the meaning and rationale for each checklist item. For each item, we include an example of good reporting and, where possible, references to relevant empirical studies and methodological literature. The PRISMA Statement, this document, and the associated Web site (http://www.prisma-statement.org/) should be helpful resources to improve reporting of systematic reviews and meta-analyses.

/pdf/the-prisma-statement-for-reporting-systematic-reviews-and-3j53e5oe33.pdf

The PRISMA Statement for Reporting Systematic Reviews and Meta-Analyses of Studies That Evaluate Health Care Interventions: Explanation and Elaboration

The Cochrane Handbook for Systematic Reviews of Interventions is the official document that describes in detail the process of preparing and maintaining Cochrane systematic reviews on the effects of healthcare interventions.

/pdf/cochrane-handbook-for-systematic-reviews-of-interventions-1xbdy6uvru.pdf

Cochrane Handbook for Systematic Reviews of Interventions

The adequacy of Fisher's approximation to the large sample variance of an intraclass correlation is investigated in the context of family studies. It is found that the approximation is highly accurate in samples of moderately large size (≧ 30 families), and can also be used for significance-testing under a broad range of circumstances. The exact sampling of distribution of the intraclass correlation coefficient is also derived.

A note on the accuracy of fisher's approximation to the large sample variance of an intraclass correlationa

A simple alternative confidence interval for the difference between two proportions.

We consider inference methods for interobserver agreement studies characterized by two raters and several outcome categories that one can naturally combine to address a series of questions of a priori interest. We propose a new method based on a series of nested, statistically independent inferences, each corresponding to a binary outcome variable obtained by combining a substantively relevant subset of the original categories. We conduct the inferences using a goodness-of-fit procedure that extends the approach of Donner and Eliasziw. The methodology presented is an alternative to methodology that places each of the outcome categories on an equal footing in estimating interobserver agreement for multinomial data. We provide two examples.

A hierarchical approach to inferences concerning interobserver agreement for multinomial data

The study of twins is widely used for research into genetic and environmental influences on human outcome measurements. For the study design in which independent samples of monozygotic and dizygotic twins are compared with respect to their similarity on a binary trait, several statistical methods have been proposed. Using a Monte Carlo simulation, we compare the five following procedures : 1) goodness-of-fit method based on the common correlation model, 2) normal approximation of the maximum likelihood estimators of the common correlation coefficients, 3) Ramakrishnan et al. [(1992) Genet Epidemiol 9 :273-282] method of odds ratio comparison, 4) generalized estimating equations method of odds ratio estimation, and 5) tetrachoric correlation method. The results show that the goodness-of-fit approach has similar or better performance in both type-one error rates and power than the other methods in all parameter settings. Its advantage with respect to type-one error rates is particularly clear under conditions of small sample sizes, extreme prevalences, or high values of the intraclass correlation coefficients. Therefore, the goodness-of-fit method is recommended for the two-sample twin study design.

Comparison of methods for analyzing binary data arising from two‐sample twin studies

Background Cluster randomization trials are widely used to test the effect of an intervention when individuals are naturally found in groups such as communities. For several separate studies of a similar intervention, it may be of interest to combine their results using meta-analysis procedures. However, this task requires consideration of both the likely dependencies among cluster members (intracluster correlation) and stratification based on the studies considered.Purpose In this article, several possible approaches for meta-analysis are considered for cluster randomization trials having a binary outcome.Methods It is first noted that the standard Mantel—Haenszel test is invalid in this context since it ignores dependencies among cluster members. Two modifications are therefore considered as well as a general inverse variance approach and a procedure based on the Woolf statistic which does not require the availability of trial-specific design effects. Empirical Type I errors and powers for the different...

Allan Donner

Papers

A note on the accuracy of fisher's approximation to the large sample variance of an intraclass correlationa

A simple alternative confidence interval for the difference between two proportions.

A hierarchical approach to inferences concerning interobserver agreement for multinomial data

Comparison of methods for analyzing binary data arising from two‐sample twin studies

Meta-analysis of community-based cluster randomization trials with binary outcomes