Cochrane Reviews have recently started including the quantity I 2 to help readers assess the consistency of the results of studies in meta-analyses. What does this new quantity mean, and why is assessment of heterogeneity so important to clinical practice? 

Systematic reviews and meta-analyses can provide convincing and reliable evidence relevant to many aspects of medicine and health care.1 Their value is especially clear when the results of the studies they include show clinically important effects of similar magnitude. However, the conclusions are less clear when the included studies have differing results. In an attempt to establish whether studies are consistent, reports of meta-analyses commonly present a statistical test of heterogeneity. The test seeks to determine whether there are genuine differences underlying the results of the studies (heterogeneity), or whether the variation in findings is compatible with chance alone (homogeneity). However, the test is susceptible to the number of trials included in the meta-analysis. We have developed a new quantity, I 2, which we believe gives a better measure of the consistency between trials in a meta-analysis.

Assessment of the consistency of effects across studies is an essential part of meta-analysis. Unless we know how consistent the results of studies are, we cannot determine the generalisability of the findings of the meta-analysis. Indeed, several hierarchical systems for grading evidence state that the results of studies must be consistent or homogeneous to obtain the highest grading.2–4

Tests for heterogeneity are commonly used to decide on methods for combining studies and for concluding consistency or inconsistency of findings.5 6 But what does the test achieve in practice, and how should the resulting P values be interpreted?

A test for heterogeneity examines the null hypothesis that all studies are evaluating the same effect. The usual test statistic …

/pdf/measuring-inconsistency-in-meta-analyses-4iysmttp08.pdf

Measuring inconsistency in meta-analyses

Users of clinical practice guidelines and other recommendations need to know how much confidence they can place in the recommendations Systematic and explicit methods of making judgments can reduce errors and improve communication We have developed a system for grading the quality of evidence and the strength of recommendations that can be applied across a wide range of interventions and contexts In this article we present a summary of our approach from the perspective of a guideline user Judgments about the strength of a recommendation require consideration of the balance between benefits and harms, the quality of the evidence, translation of the evidence into specific circumstances, and the certainty of the baseline risk It is also important to consider costs (resource utilisation) before making a recommendation Inconsistencies among systems for grading the quality of evidence and the strength of recommendations reduce their potential to facilitate critical appraisal and improve communication of these judgments Our system for guiding these complex judgments balances the need for simplicity with the need for full and transparent consideration of all important issues

/pdf/grading-quality-of-evidence-and-strength-of-recommendations-4juchvwoau.pdf

Grading quality of evidence and strength of recommendations.

https://www.just.edu.jo/DIC/ClinicGuidlines/Management%20of%20hip%20and%20knee%20Osteoarthritis.pdf

OARSI recommendations for the management of hip and knee osteoarthritis, Part II: OARSI evidence-based, expert consensus guidelines

To comprehend the results of a randomized controlled trial (RCT), readers must understand its design, conduct, analysis and interpretation. That goal can only be achieved through complete transparency from authors. Despite several decades of educational efforts, the reporting of RCTs needs improvement. Investigators and editors developed the original CONSORT (Consolidated Standards of Reporting Trials) statement to help authors improve reporting by using a checklist and flow diagram. The revised CONSORT statement presented in this paper incorporates new evidence and addresses some criticisms of the original statement.

http://www.people.vcu.edu/~albest/Guidance/CONSORTjama.pdf

The CONSORT statement: revised recommendations for improving the quality of reports of parallel group randomized trials

Scottish Intercollegiate Guidelines Network

The Scottish Intercollegiate Guidelines Network (SIGN) develops evidence based clinical guidelines for the NHS in Scotland. The key elements of the methodology are (a) that guidelines are developed by multidisciplinary groups; (b) they are based on a systematic review of the scientific evidence; and (c) recommendations are explicitly linked to the supporting evidence and graded according to the strength of that evidence.

Until recently, the system for grading guideline recommendations was based on the work of the US Agency for Healthcare Research and Quality (formerly the Agency for Health Care Policy and Research). 1 2 However, experience over more than five years of guideline development led to a growing awareness of this system's weaknesses. Firstly, the grading system was designed largely for application to questions of effectiveness, where randomised controlled trials are accepted as the most robust study design with the least risk of bias in the results. However, in many areas of medical practice randomised trials may not be practical or ethical to undertake; and for many questions other types of study design may provide the best evidence. Secondly, guideline development groups often fail to take adequate account of the methodological quality of individual studies and the overall picture presented by a body of evidence rather than individual studies or they fail to apply sufficient judgment to the overall strength of the evidence base and its applicability to the target population of the guideline. Thirdly, guideline users are often not clear about the implications of the grading system. They misinterpret the grade of recommendation as relating to its importance, rather than to the strength of the supporting evidence, and may therefore fail to give due weight to low grade recommendations.

#### Summary points

A revised system of determining levels of evidence and grades …

https://www.bmj.com/content/bmj/323/7308/334.1.full.pdf

Juliet Miller

Papers

A new system for grading recommendations in evidence based guidelines