Manuscript Published in Information and Software Technology 55 (2013) 2049-2075
1
A Systematic Review of Systematic Review Process Research in
Software Engineering
Barbara Kitchenham and Pearl Brereton
School of Computing and Mathematics
Keele University
Staffordshire ST5 5BG
{b.a.kitchenham,o.p.brereton}@keele.ac.uk
Abstract
Context: Many researchers adopting systematic reviews (SRs) have also published
papers discussing problems with the SR methodology and suggestions for improving it.
Since guidelines for SRs in software engineering (SE) were last updated in 2007, we
believe it is time to investigate whether the guidelines need to be amended in the light of
recent research.
Objective: To identify, evaluate and synthesize research published by software
engineering researchers concerning their experiences of performing SRs and their
proposals for improving the SR process.
Method: We undertook a systematic review of papers reporting experiences of
undertaking SRs and/or discussing techniques that could be used to improve the SR
process. Studies were classified with respect to the stage in the SR process they
addressed, whether they related to education or problems faced by novices and whether
they proposed the use of textual analysis tools.
Results: We identified 68 papers reporting 63 unique studies published in SE
conferences and journals between 2005 and mid-2012. The most common criticisms of
SRs were that they take a long time, that SE digital libraries are not appropriate for broad
literature searches and that assessing the quality of empirical studies of different types is
difficult.
Conclusion: We recommend removing advice to use structured questions to construct
search strings and including advice to use a quasi-gold standard based on a limited
manual search to assist the construction of search stings and evaluation of the search
process. Textual analysis tools are likely to be useful for inclusion/exclusion decisions
and search string construction but require more stringent evaluation. SE researchers
would benefit from tools to manage the SR process but existing tools need independent
validation. Quality assessment of studies using a variety of empirical methods remains a
major problem.
Keywords: systematic review; systematic literature review; systematic review
methodology; mapping study.
1. Introduction
In 2004 and 2005, Kitchenham, Dybå and Jørgensen proposed the adoption of evidence-
based software engineering (EBSE) and the use of systematic reviews of the software
engineering literature to support EBSE (Kitchenham et al., 2004 and Dybå et al., 2005).
Since then, systematic reviews (SRs) have become increasingly popular in empirical
software engineering as demonstrated by three tertiary studies reporting the numbers of
2
[Type text]
2
[Type text]
2
[Type text]
such studies (Kitchenham et al., 2009, Kitchenham et al., 2010a, da Silva et al, 2011).
Many of these studies adopted the guidelines for undertaking systematic review, based on
medical standards, proposed by Kitchenham (2004), and revised first by Biolchini et al
(2005) to take into account practical problems associated with using the guidelines and
later by Kitchenham and Charters (2007) who incorporated approaches to systematic
reviews proposed by sociologists.
As software engineers began to use the SR technology, many researchers also began to
comment on the SR process itself. Brereton et al (2007) wrote one of the first papers that
commented on issues connected with performing SRs and many such papers have
followed since, for example:
Staples and Niazi (2006, 2007) discussed the issues they faced extracting and
aggregating qualitative information.
Budgen et al (2008) and Petersen et al (2008) identified the difference between
mapping studies and conventional systematic reviews.
Kitchenham et al. (2010c) considered the use of SRs and mapping studies in an
educational setting
MacDonnell et al. (2010) and Kitchenham et al. (2011) studied the claims of the
SR technology with respect to reliability/consistency
Dieste and Padua (2007) and Skoglund and Runeson (2009) investigated how to
improve the search process
Kitchenham et al. (2010b) investigated how best to evaluate the quality of
primary studies (i.e. the empirical studies found by the systematic review search
and selection process).
It therefore seems appropriate to identify the current status of such studies in software
engineering, and identify whether there is evidence for revising and/or extending the
guidelines for performing systematic reviews in software engineering. To that end we
undertook a systematic review of papers that discuss problems with the current SR
guidelines and/or propose methods to address those problems.
Section 2 discusses the aims of our research, reports related research and identifies the
specific research questions we address. Section 3 reports the search and paper selection
process we adopted and reports the basic limitations of our approach. Section 4 reports
the outcome of our search and selection process and its validity. We also report the
reliability of our data extraction and quality assessment process. Section 5 presents our
aggregation and synthesis of information from the papers we included in the study.
Section 6 discusses our results and the limitations that arose during our study. We present
our conclusions in section 7.
2. Aims and Research Questions
Our aim is to assess whether our guidelines for performing systematic reviews in
software engineering need to be amended to reflect the results of methodological
investigations of SRs undertaken by software engineering researchers. In order to do this
Manuscript Published in Information and Software Technology 55 (2013) 2049-2075
3
we undertook a systematic review of papers reporting experiences of using the SR
methodology and/or investigating the SR process in software engineering (SE). We use
this information to assess whether SRs have delivered the expected benefits to SE, to
identify problems found by software engineering researchers when undertaking SRs, and
to identify and assess proposals aimed at addressing perceived problems with the SR
methodology.
There have been two mapping studies that address methods for supporting SRs. Felizardo
et al (2012) report a mapping study of the use of visual data mining (VDM) techniques to
support SRs. Their mapping study concentrated on a specific technique and was not
restricted to SE studies. In contrast, our SR considers a broader range of techniques but is
restricted to studies in the SE domain. Marshall and Brereton (2013) have undertaken a
mapping study of tools to support SRs in SE. Compared with our study:
Their mapping study focused specifically on tools for SRs in the SE.
They used a search string-based automated search process, using papers identified
in this study as a set of known studies to refine their search strings.
The time period of their search was longer, going from 2005 to the end of 2012.
Thus the value of this study is that it addresses a wider range of technologies than either
of the mapping studies, and as an SR provides a more in-depth aggregation of the results
of the identified primary studies.
Our SR addresses the following research questions:
RQ1. What papers report experiences of using the SR methodology and/or
investigate the SR process in software engineering between the years 2005 and 2012
(to June)?
RQ2. To what extent has research confirmed the claims of the SR methodology?
RQ3. What problems have been observed by SE researchers when undertaking SRs?
RQ4. What advice and/or techniques related to performing SR tasks have been
proposed and what is the strength of evidence supporting them?
3. Search and Selection Process
Before starting our SR, we produced a review protocol which is summarised in this
section. Figures 1, 2 and 3 give an overview of the search and selection process which are
described in more detail below.
3.1 Initial search process
Kitchenham undertook an initial informal search of two conference proceedings
(Evaluation and Assessment in Software engineering and Empirical Software
Engineering and Measurement) from 2005 to mid 2012 which together with personal
knowledge identified 55 papers related to methods for performing systematic reviews and
mapping studies in SE. This initial search confirmed that there are a substantial number
of papers on the topic and that a systematic review would be appropriate. It also provided
the information needed to guide the manual search process.
4
[Type text]
4
[Type text]
4
[Type text]
3.2 Search and Selection Process
3.2.1 Stage 1 Manual Search and Selection
The 55 known papers identified the main sources of papers on methodology to be:
Evaluation and Assessment in Software Engineering (EASE): 21 papers
Empirical Software Engineering and Measurement (ESEM): 18 papers
Information and Software Technology (IST): 6 papers
Empirical software engineering journal (ESE): 2 papers
Journal of Systems and Software (JSS): 2 papers
International Conference on Software Engineering (ICSE): 2 papers
Five other sources each published a single SR methodology paper:
Empirical Assessment for Software Technologies (EAST)
Advanced Engineering Informatics
IEEE Transactions of Software Engineering (TSE)
Lecture notes on Computer Science Volume 5089
Proceedings of Psychology of Programming Special Interest Group (PPIG) ’08.
Of these sources only EAST which is targeted at evidence-based software engineering
and systematic reviews was both relevant and unlikely to be found by an automated
search. Kitchenham attended EAST 2012 and identified relevant papers at the workshop.
We both undertook an independent manual search of the main sources from 2005 to June
2012 (with ESEM 2012 being searched using the published program) and classified each
paper as included or excluded. The emphasis of the manual search was on including
papers unless they were clearly irrelevant. The results of the two searches were collated
and any papers we disagreed about were read and then discussed. If we could not come to
an agreement about a paper we classified it as “include”.
3.2.2 Stage 1 Citation-based Search and Selection
To support the manual search, an automated search based on citation analysis (also
known as forward snowballing) was performed. Kitchenham searched SCOPUS for all
papers referencing the following papers:
Kitchenham, B.A.; S. (2007) Charters, Guidelines for performing systematic
literature reviews in software engineering. (Search date 25
th
June 2012)
Kitchenham, B. (2004) Procedures for undertaking systematic reviews.
Kitchenham, B., Tore Dybå and Magne Jørgensen. (2004) Evidence-based
Software Engineering. ICSE. (Search date 25
th
June 2012)
Dybå, Tore; Barbara Kitchenham, and Magne Jørgensen. (2005) Evidence-based
Software Engineering for Practitioners, IEEE Software. (Search date 25
th
June
2012)
Manuscript Published in Information and Software Technology 55 (2013) 2049-2075
5
Brereton, Pearl; Barbara A. Kitchenham, David Budgen, Mark Turner, Mohamed
Khalil (2007) Lessons from applying the systematic literature review process
within the software engineering domain. (Search date 29
th
June 2012)
After removing duplicates, we both evaluated each paper for inclusion in the set of
candidate papers based on title and abstract. The main emphasis was to include papers
unless they were clearly irrelevant. The decisions of each author were collated. Papers
which both authors agreed to include were included and any papers which both authors
agreed to exclude were excluded. Any papers for which the inclusion/exclusion
assessment differed between authors were discussed until either agreement was reached
or the paper was provisionally included.
