http://www.jmde.com/ Book Reviews
Journal of MultiDisciplinary Evaluation, Volume 6, Number 14
ISSN 1556-8180
August 2010
143
Cochrane Handbook for Systematic Reviews of
Interventions Edited by Julian P. T. Higgins &
Sally Green, 2008. Chicester, West Sussex, UK:
Wiley-Blackwell. $42.40
Reviewed by Michele Tarsilla
Western Michigan University
t is always good news when an
evaluation report attests either the
effectiveness of a medical therapy or the
sustainability of a public program whose
development or implementation have
been funded by the government (both at
the federal and state level). That the
expected goals of an after-school program
or the envisaged effects of a newly
approved diabetes treatment have been
fully attained and that, as a result,
taxpayers’ money was wisely spent, are
especially relevant from an accountability
perspective, even more so in times of
financial distress, like the one we are
currently living in. Learning about
programs’ effectiveness is certainly good
but it is not sufficient. In other words,
demonstrating that Project A
accomplished all its objectives and that all
its effects were statistically significant
does not constitute evidence of
substantive impact. What would be more
beneficial instead, both from a
programmatic and scientific research
standpoints, is comparing the effect size
of Project A with that of Projects B and C.
However, for studies to be juxtaposed
based on the estimation of their effect
size, it is necessary that they be directly
comparable with each other. However, for
that to happen, researchers will need not
only to employ equivalent outcome
variables but also to measure them
consistently with the scientific literature
available on their specific topic of interest.
That is exactly where one of the meta-
analysis’ added value lies: enhancing the
uniformization of outcome indicators and
impact measurements across a variety of
studies in a given field (Hedges et al,
1988).
Introduced in the late 1970’s (Smith &
Glass, 1977), meta-analysis mainly
consists in the selection and analysis of
quantitative studies meeting the three
following criteria: (1) the display of
similar substantive, methods and/or
extrinsic characteristics (Lipsey 1994); (2)
the inclusion of identical outcomes of
interest; and 3) the presentation of
comparable effect sizes (Rosenthal, 2000;
Raudenbush & Bryk, 2002; Shadish,
Cook, & Campbell, 2002). Thanks to
I
Michele Tarsilla
Journal of MultiDisciplinary Evaluation, Volume 6, Number 14
ISSN 1556-8180
August 2010
144
meta-analysis, a body of comparable
information on the effectiveness of
policies or programs accumulates and is
thus ready to be used as the basis for
decision-making as well as for the setting
of future research agenda (Howard et al,
2000). Quite a fascinating topic if it were
not for the fact that meta-analysis is not of
immediate fruition to the most, mainly
due to the statistical complexity and
highly technical content of the research
topics associated with it. While peer-
reviewed articles on meta-analysis have
contributed to the dissemination of this
research method within academic circles
over the last twenty years, a manual on
this topic that could make meta-analysis
more accessible among the general public
(and that could integrate some of the most
recent ideas and trends into the current
discourse on synthesis of quantitative
research) has been missing for over a
decade.
The paucity of specialized manuals or
how-to-guides on meta-analysis then
explains why the publication of the
Cochrane Handbook for Systematic
Reviews of Interventions edited by Julian
P. T. Higgins and Sally Green was
particularly welcome. In particular, this
book is an in-depth description of the
Cochrane approach (www.cochrane.org),
a very rigorous and somewhat expensive
methodology for conducting systematic
reviews of interventions. Created in 1993
and already consisting of more than
15,000 contributions in over 100
countries. As an evaluator currently
completing his doctoral studies in
interdisciplinary evaluation, I started
reading it with great trepidation myself.
Ever since I heard about it in one of the
design classes during my first semester in
the program, I thought that this reading
was a must for any student in evaluation
or research methods. As the book was also
recommended to me by a colleague
working at the World Bank on the
evaluation of HIV and AIDS programs in
sub-Saharan Africa and Latin America, I
convinced myself quite rapidly that this
Handbook would also be of particular
interest for researchers and evaluators
working in international development.
Although I had already had the
opportunity to skim through this
somewhat intimidating book (a total of
646 pages divided in three main parts and
22 chapters) at the beginning of the year
(I had already found it quite a useful
reading then), I went back to it with an
even greater interest more recently, once I
was invited to write a review for it.
The intellectual and programmatic
benefits associated with the conduct of
meta-analysis certainly influenced my
decision to write a review of this book: I
personally engaged myself in the
systematic review of street children
interventions in the Democratic Republic
of Congo and community-based HIV and
AIDS programs in Kenya. However, on a
more practical level, I was particularly
drawn to meta-analysis by the facility of
implementing it from my own desk
without having to travel overseas, as I
have been doing over the last eight years.
Going back to a more thorough read of
this book a second time was quite
enlightening for five main reasons. First,
the clarity of language and the cohesive
presentation of topics throughout the
whole book. I particularly appreciated the
preliminary listing of key points at the
beginning of each chapter as well as the
summary of findings at the end. That was
especially true for the first two parts of the
book. In particular, Part I introduces the
Cochrane approach and provides the
reader with a step-by-step guide on how
to plan and prepare a systematic review;
Part II offers a more in-depth description
Michele Tarsilla
Journal of MultiDisciplinary Evaluation, Volume 6, Number 14
ISSN 1556-8180
August 2010
145
of the meta-analytical methodology,
including the question and inclusion
criteria development (questions should be
describing participants, interventions,
comparisons and outcomes), the search,
the data collection and the data analysis.
Second, the wealth of practical
information provided to reader for the
identification and inclusion of appropriate
studies in their meta-analytical endeavors.
Readers would especially appreciate the
list of bibliographic databases
recommended for the search of trails
report in Chapter 6, such as the Health
InterNetwork Access to Research
Initiative (HINAEY) or the International
Network for the Availability of Scientific
Publications (INASP) and the Cochrane
Central Register of Controlled Trials
(CENTRAL), the African Medicus Index,
the Australasian Medical Index, or the
Allied Complementary Medicine (AMED).
In the same chapter, readers will also
learn that it is preferable to (1) conduct
searches from 2005 onwards when
searching in MEDLINE; (2) conduct
searchers for the most recent two years
when searching in EMBASE; (3) truncate
the search term to capture a larger
number of articles or reports on the topic
of research interest; and (4) add the term
NEXT to the search word to incorporate
either auto-pluralization or auto-
singularization in the search.
Third, the authors have been very
thorough in addressing not only
methodological but also practical issues
which any researcher conducting a review
will encounter throughout the whole
search process: how to deal with the issue
of sensitivity (the number of relevant
report identified over the total of existing
studies) and precision (the number of
relevant reports divided by the number of
identified reports); how to reduce the
selection bias; what type of the term to
search for in databases; the overall
duration of the search process or the type
of bibliographic software to use in order to
save the references of all the reports
included in the final meta-analysis report.
In Chapter 8, for instance, readers will
learn how to deal with the risks of bias
and will also be provided with a sort of
checklist on possible validity threats and
corresponding actions to take in order to
successfully address them (real or not).
Next, Chapter 9 reiterates the advantages
of conducting a meta-analysis: increased
power, improved precision, demonstrated
consistent effect of an intervention across
several settings and populations,
generation of new hypothesis and
settlement of controversies on apparently
conflicting studies.
Fourth, the book provides a
comprehensive overview of issues which
any researcher, regardless of the specific
approach adopted, should be familiar
with. For example, the reader is reminded
that, although case studies represent the
least rigorous type of study to include in a
systematic review, these are normally the
ones determining the removal of drugs
from the market (Glasziou, 2007).
Likewise, the references to some of the
approach main components (e.g., the
rating system used by the Cochrane
approach to assess the quality of studies,
also known under the name of GRADE)
leads the reader to explain the supremacy
of RCT based on the role played by them
is such collaborative research efforts and
corresponding methodologies, as the
Cochrane.
Fifth, the authors made a specific
effort to touch upon specific and
controversial topics, such as the use of
qualitative data and evidence collected
through non-randomized trials (e.g., in
case the health intervention does not lend
itself to being randomly assigned or just a
Michele Tarsilla
Journal of MultiDisciplinary Evaluation, Volume 6, Number 14
ISSN 1556-8180
August 2010
146
way to supplement existing randomized
trial evidence), the presentation of
adverse outcomes or the needed
adjustments in case of missing data.
The book is definitely worth reading
for anyone interested in learning not only
about the Cochrane approach but also
about research methods more in general.
That being said, the book also presents
four major shortcomings.
First, the missed opportunity to reach
out to a larger audience. Meta-analysis
strongly relies on the review of unbiased
studies which also employ identical
outcomes measures. However, such
studies are particularly hard to come by in
areas other than medicine (it is not
surprising that MEDLINE and EMBASE
are the two databases including the
largest number of randomized controlled
trials). In particular, the authors seem to
have missed to elaborate on the utility of
the methodology and its direct
applicability across a much wider variety
of areas than merely medicine. Rather,
than targeting researchers in the medical
field, the authors should have made the
content of the book to a more diverse
crowd, by drawing for instance on
examples of meta-analysis conducted in
education and nutrition. That could have
allowed the authors to initiate researchers
to the use of this approach on a small
scale within their own domain of interest
and lay the ground for a closer
coordination at a later stage.
Second, the impression that one
receives by reading this book is that the
approach described in it will never be
implemented by readers independently,
that is, without Cochrane technical
support. In particular, Cochrane
methodology needs to be complied with
quite strictly before a review could be
certified as a Cochrane. To this purpose, a
global team of experts (fifty-two Review
Groups, several Methods Groups,
dedicated Trial Search Co-ordinators and
Cochrane Centers) have been put in place
to manage the editorial process of
publishing of both protocols and reviews.
That seems to be in conflict with the
instructional purpose which a Handbook
like this should to pursue. Put simply, this
book purports to divulgate knowledge
about the approach but it constantly
reminds readers that they will never be
able to replicate it on their own without
the support of highly paid specialists
trained in this specific methodology.
Thus, the use of the methodology is
reserved to those who have the both the
time to conduct and maintain the review
as well as the funds to cover its high costs
of implementation. Quite a controversial
issue, if one takes into account that a
meta-analysis, first and foremost, intends
to (1) build a body of knowledge on
effective programs and policies and (2)
make its findings easily available to the
largest possible number of people.
Therefore, the question is whether the
elevated costs for implementing the
Cochrane approach does not ultimately
represent a disincentive to promote the
spread of meta-analytical thinking (Kline,
2004). Interestingly enough, although the
level of compliance with Cochrane
guidelines is quite strict when conducting
systematic reviews, authors are still
advised to add in their peer-review
articles a disclaimer about their review
not reflecting the Cochrane
Collaboration’s opinions.
Third, readers need to be aware that
the implementation of the guidelines
presented in this handbook is not suitable
for conducting a meta-analysis across all
types of interventions (as the title might
mislead one to believe). As mentioned in
Chapter 1 and as the origin of the
approach name suggests (Archie
Michele Tarsilla
Journal of MultiDisciplinary Evaluation, Volume 6, Number 14
ISSN 1556-8180
August 2010
147
Cochrane was a very well know
epidemiologist in the United Kingdom),
the Cochrane approach is more directly
applicable to the review of healthcare
interventions only. As a result, most of the
studies included in Cochrane reviews will
be randomized trials. That being said,
several chapters of the book might be of
great interest to a larger audience (e.g.,
the chapters on addressing reporting
biases, detecting bias risks and presenting
analysis results). The idea of collaboration
(not just decision-makers but also
consumers are involved in the peer-review
process) is also one that could be
immediate replicated to a vast array of
field.
Fourth, the excessive search for
clinical objectivity represents a detriment
to the easy application of the analysis
findings. In one of the passages, it is
suggested that “authors of Cochrane
reviews should not make
recommendations” (p. 380). That is quite
puzzling, especially given the highly
participatory and collaborative peer-
review process associated with the
Cochrane Approach. In particular, I
wonder why, while researchers and
consumers are explicitly engaged in the
peer-review process, health professionals
and clinical practical guideline developers
are left out of the general discussion,
mainly with respect to the utilization of
findings. Before the publication of the
review, for instance, the provision of
concrete recommendations or suggestions
for readers would be quite useful,
especially for those health authorities,
health managers or policy-makers with
relatively weak analytical and
interpretative skills.
In conclusion, despite the editors’
fallacies with respect to the targeting and
envisaged applicability of the book’s
content, I strongly recommend the
reading of it to both students and
practitioners with a particular interest in
research methods. In doing so, I would
remind readers of a caveat, though: only
by stretching some of the content or
related examples presented in the book
beyond the medical field, the utility of the
Cochrane meta-analytical approach will
outweigh the statistical complexity and
the stringent implementation conditions
associated with it.
References
Glasziou P., Chalmers, I., Rawlins M., &
McCulloch, P. (2007). When are
randomized trials unnecessary?
Picking signal from noise. British
Medical Journal, 334, 349-351
Hedges, L. V., & Vevea, J. L. (1998). Fixed
and random effects models in meta-
analysis. Psychological Methods, 4,
486-504
Howard, G. S., Maxwell, S. E., & Fleming,
K. G. (2000). The proof of the
pudding: An illustration of the relative
strengths of null hypothesis, meta-
analysis, and Bayesian analysis.
Psychological Methods, 5, 315-332.
Kline, R. B. (2004). Beyond significance
testing: Reforming data analysis
methods in behavioral research.
Washington DC: American
Psychological Association.
Lipsey, M. W. (1994). Identifying
potentially interesting variables and
analysis opportunities. In H. Cooper &
L. V. Hedges (Eds.), The handbook of
research synthesis (pp. 111-124). New
York: Russel Sage Foundation.
Raudenbush, S. W., & Bryk, A. S. (2002).
Hierarchical linear models (2
nd
ed.).
Thousand Oaks, CA: Sage.
Rosenthal, R., Rosnow, R. L., & Rubin, D.
B. (2000). Contrasts and effect sizes