Research synthesis in software engineering
TL;DR: It is concluded that, despite the focus on systematic reviews, there is limited attention paid to research synthesis in software engineering and a repertoire of synthesis methods needs to be an integral part of systematic reviews to increase their significance and utility for research and practice.
Abstract: ContextComparing and contrasting evidence from multiple studies is necessary to build knowledge and reach conclusions about the empirical support for a phenomenon. Therefore, research synthesis is at the center of the scientific enterprise in the software engineering discipline. ObjectiveThe objective of this article is to contribute to a better understanding of the challenges in synthesizing software engineering research and their implications for the progress of research and practice. MethodA tertiary study of journal articles and full proceedings papers from the inception of evidence-based software engineering was performed to assess the types and methods of research synthesis in systematic reviews in software engineering. ResultsAs many as half of the 49 reviews included in the study did not contain any synthesis. Of the studies that did contain synthesis, two thirds performed a narrative or a thematic synthesis. Only a few studies adequately demonstrated a robust, academic approach to research synthesis. ConclusionWe concluded that, despite the focus on systematic reviews, there is limited attention paid to research synthesis in software engineering. This trend needs to change and a repertoire of synthesis methods needs to be an integral part of systematic reviews to increase their significance and utility for research and practice.
Citations
More filters
Book•
16 Jun 2012
TL;DR: The purpose of Experimentation in Software Engineering is to introduce students, teachers, researchers, and practitioners to empirical studies in software engineering, using controlled experiments, and provides indispensable information regarding empirical Studies in particular for experiments, but also for case studies, systematic literature reviews, and surveys.
Abstract: Like other sciences and engineering disciplines, software engineering requires a cycle of model building, experimentation, and learning. Experiments are valuable tools for all software engineers who are involved in evaluating and choosing between different methods, techniques, languages and tools. The purpose of Experimentation in Software Engineering is to introduce students, teachers, researchers, and practitioners to empirical studies in software engineering, using controlled experiments. The introduction to experimentation is provided through a process perspective, and the focus is on the steps that we have to go through to perform an experiment. The book is divided into three parts. The first part provides a background of theories and methods used in experimentation. Part II then devotes one chapter to each of the five experiment steps: scoping, planning, execution, analysis, and result presentation. Part III completes the presentation with two examples. Assignments and statistical material are provided in appendixes. Overall the book provides indispensable information regarding empirical studies in particular for experiments, but also for case studies, systematic literature reviews, and surveys. It is a revision of the authors book, which was published in 2000. In addition, substantial new material, e.g. concerning systematic literature reviews and case study research, is introduced. The book is self-contained and it is suitable as a course book in undergraduate or graduate studies where the need for empirical studies in software engineering is stressed. Exercises and assignments are included to combine the more theoretical material with practical aspects. Researchers will also benefit from the book, learning more about how to conduct empirical studies, and likewise practitioners may use it as a cookbook when evaluating new methods or techniques before implementing them in their organization.
2,079 citations
Cites background from "Research synthesis in software engi..."
...For more detail, refer to Cruzes and Dybå [39] and related references....
[...]
...Cruzes and Dybå present an overview of qualitative synthesis methods [39]....
[...]
...Cruzes and Dybå [39] surveyed secondary studies in software engineering, which included synthesis of empirical evidence....
[...]
...Cruzes and Dybå present an overview of qualitative synthesis methods [39]....
[...]
TL;DR: There was a need to provide an update of how to conduct systematic mapping studies and how the guidelines should be updated based on the lessons learned from the existing systematic maps and SLR guidelines.
Abstract: Context Systematic mapping studies are used to structure a research area, while systematic reviews are focused on gathering and synthesizing evidence. The most recent guidelines for systematic mapping are from 2008. Since that time, many suggestions have been made of how to improve systematic literature reviews (SLRs). There is a need to evaluate how researchers conduct the process of systematic mapping and identify how the guidelines should be updated based on the lessons learned from the existing systematic maps and SLR guidelines. Objective To identify how the systematic mapping process is conducted (including search, study selection, analysis and presentation of data, etc.); to identify improvement potentials in conducting the systematic mapping process and updating the guidelines accordingly. Method We conducted a systematic mapping study of systematic maps, considering some practices of systematic review guidelines as well (in particular in relation to defining the search and to conduct a quality assessment). Results In a large number of studies multiple guidelines are used and combined, which leads to different ways in conducting mapping studies. The reason for combining guidelines was that they differed in the recommendations given. Conclusion The most frequently followed guidelines are not sufficient alone. Hence, there was a need to provide an update of how to conduct systematic mapping studies. New guidelines have been proposed consolidating existing findings.
1,598 citations
Cites background or methods from "Research synthesis in software engi..."
...RQ3, Venues: Around half of the studies are published in conferences and workshops, and the other half in reputable journals....
[...]
...Identify objective criteria for decision [80,20,45,37] Add additional reviewer, resolve disagreements between them when needed [73,64,76,41,6,30,80,20, 72,33,60,77,45,85,86,13,37]...
[...]
...Venues of publication (RQ3) In this study, peer-reviewed venues (including journals, as well as peer-reviewed conferences and workshops) were considered....
[...]
...Journal [72,33,12,34,74,35,36,13,76,37,64,22,20,54,38,80,30,81,21, 85,55,53,23] Conference [69,3,60,70,63,75,4,77,42,58,32,78,6,43,39,44,82–...
[...]
...Rank Venue Studies 1 Information & Software Technology (IST) [23,33,12,34– 36,13,37,22,20,38,30] 2 Evaluation and Assessment in Software Engineering (EASE) [39–46] 3 Empirical Software Engineering and Measurement (ESEM) [32,5,3,47] Fig....
[...]
Journal Article•
TL;DR: This book will be essential reading for all those who loved (or loathed) the arguments developed in Realistic Evaluation and offers a complete blueprint for research synthesis, supported by detailed illustrations and worked examples from across the policy waterfront.
Abstract: Author Ray Pawson presents a devastating critique of the dominant approach to systematic review namely the 'meta-analytic' approach as sponsored by the Cochrane and Campbell collaborations. In its place is commended an approach that he terms 'realist synthesis'. On this vision, the real purpose of systematic review is better to understand program theory, so that policies Author Ray Pawson presents a devastating critique of the dominant approach to systematic review namely the 'meta-analytic' approach as sponsored by the Cochrane and Campbell collaborations. In its place is commended an approach that he terms 'realist synthesis'. On this vision, the real purpose of systematic review is better to understand program theory, so that policies can be properly targeted and developed to counter an ever-changing landscape of social problems. The book will be essential reading for all those who loved (or loathed) the arguments developed in Realistic Evaluation (Sage, 1997). It offers a complete blueprint for research synthesis, supported by detailed illustrations and worked examples from across the policy waterfront.
1,037 citations
TL;DR: Although there are a set of fault prediction studies in which confidence is possible, more studies are needed that use a reliable methodology and which report their context, methodology, and performance comprehensively.
Abstract: Background: The accurate prediction of where faults are likely to occur in code can help direct test effort, reduce costs, and improve the quality of software. Objective: We investigate how the context of models, the independent variables used, and the modeling techniques applied influence the performance of fault prediction models. Method: We used a systematic literature review to identify 208 fault prediction studies published from January 2000 to December 2010. We synthesize the quantitative and qualitative results of 36 studies which report sufficient contextual and methodological information according to the criteria we develop and apply. Results: The models that perform well tend to be based on simple modeling techniques such as Naive Bayes or Logistic Regression. Combinations of independent variables have been used by models that perform well. Feature selection has been applied to these combinations when models are performing particularly well. Conclusion: The methodology used to build models seems to be influential to predictive performance. Although there are a set of fault prediction studies in which confidence is possible, more studies are needed that use a reliable methodology and which report their context, methodology, and performance comprehensively.
1,012 citations
Cites background from "Research synthesis in software engi..."
...Synthesizing findings across studies is notoriously difficult and many software engineering SLRs have been shown to present no synthesis [13]....
[...]
22 Sep 2011
TL;DR: This paper conceptualizes the thematic synthesis approach in software engineering as a scientific inquiry involving five steps that parallel those of primary research.
Abstract: Thematic analysis is an approach that is often used for identifying, analyzing, and reporting patterns (themes) within data in primary qualitative research. 'Thematic synthesis' draws on the principles of thematic analysis and identifies the recurring themes or issues from multiple studies, interprets and explains these themes, and draws conclusions in systematic reviews. This paper conceptualizes the thematic synthesis approach in software engineering as a scientific inquiry involving five steps that parallel those of primary research. The process and outcome associated with each step are described and illustrated with examples from systematic reviews in software engineering.
634 citations
References
More filters
Book•
01 Dec 1969TL;DR: The concepts of power analysis are discussed in this paper, where Chi-square Tests for Goodness of Fit and Contingency Tables, t-Test for Means, and Sign Test are used.
Abstract: Contents: Prefaces. The Concepts of Power Analysis. The t-Test for Means. The Significance of a Product Moment rs (subscript s). Differences Between Correlation Coefficients. The Test That a Proportion is .50 and the Sign Test. Differences Between Proportions. Chi-Square Tests for Goodness of Fit and Contingency Tables. The Analysis of Variance and Covariance. Multiple Regression and Correlation Analysis. Set Correlation and Multivariate Methods. Some Issues in Power Analysis. Computational Procedures.
115,069 citations
"Research synthesis in software engi..." refers background or methods in this paper
...Following the post hoc method, S6 aggregated the power of each test in the primary studies in relation to Cohen’s definitions of small, medium, and large effect sizes [6]....
[...]
...Examples of these methods references included Noblit and Hare’s meta-ethnography [39], Ragin’s qualitative comparative method [45], Miles and Huberman’s methods [36], Strauss and Corbin’s constant comparison method [8], and Cohen’s post hoc power calculations [6]....
[...]
...[5] B.P. Cohen, Developing Sociological Knowledge: Theory and Method, second Ed., Nelson-Hall, Chicago, 1989....
[...]
...Post-hoc power calculations S6 Cohen [6] (S6) Yes...
[...]
...[6] J. Cohen, Statistical Power Analysis for the Behavioral Sciences, second ed., Laurence Erlbaum, 1988....
[...]
TL;DR: Thematic analysis is a poorly demarcated, rarely acknowledged, yet widely used qualitative analytic method within psychology as mentioned in this paper, and it offers an accessible and theoretically flexible approach to analysing qualitative data.
Abstract: Thematic analysis is a poorly demarcated, rarely acknowledged, yet widely used qualitative analytic method within psychology. In this paper, we argue that it offers an accessible and theoretically flexible approach to analysing qualitative data. We outline what thematic analysis is, locating it in relation to other qualitative analytic methods that search for themes or patterns, and in relation to different epistemological and ontological positions. We then provide clear guidelines to those wanting to start thematic analysis, or conduct it in a more deliberate and rigorous way, and consider potential pitfalls in conducting thematic analysis. Finally, we outline the disadvantages and advantages of thematic analysis. We conclude by advocating thematic analysis as a useful and flexible method for qualitative research in and beyond psychology.
103,789 citations
"Research synthesis in software engi..." refers methods in this paper
...Thematic analysis/synthesis [3] Thematic analysis is a method for identifying, analyzing, and reporting patterns (themes) within data....
[...]
Book•
12 Oct 2017
TL;DR: The Discovery of Grounded Theory as mentioned in this paper is a book about the discovery of grounded theories from data, both substantive and formal, which is a major task confronting sociologists and is understandable to both experts and laymen.
Abstract: Most writing on sociological method has been concerned with how accurate facts can be obtained and how theory can thereby be more rigorously tested. In The Discovery of Grounded Theory, Barney Glaser and Anselm Strauss address the equally Important enterprise of how the discovery of theory from data--systematically obtained and analyzed in social research--can be furthered. The discovery of theory from data--grounded theory--is a major task confronting sociology, for such a theory fits empirical situations, and is understandable to sociologists and laymen alike. Most important, it provides relevant predictions, explanations, interpretations, and applications. In Part I of the book, "Generation Theory by Comparative Analysis," the authors present a strategy whereby sociologists can facilitate the discovery of grounded theory, both substantive and formal. This strategy involves the systematic choice and study of several comparison groups. In Part II, The Flexible Use of Data," the generation of theory from qualitative, especially documentary, and quantitative data Is considered. In Part III, "Implications of Grounded Theory," Glaser and Strauss examine the credibility of grounded theory. The Discovery of Grounded Theory is directed toward improving social scientists' capacity for generating theory that will be relevant to their research. While aimed primarily at sociologists, it will be useful to anyone Interested In studying social phenomena--political, educational, economic, industrial-- especially If their studies are based on qualitative data.
53,267 citations
Additional excerpts
...Grounded theory [8,21] Grounded theory is a primary research approach that describes methods for qualitative sampling, data collection, and data analysis....
[...]
44,847 citations
20,446 citations
"Research synthesis in software engi..." refers methods in this paper
...Grounded theory [8,21] Grounded theory is a primary research approach that describes methods for qualitative sampling, data collection, and data analysis....
[...]
..., [8])....
[...]
...Examples of these methods references included Noblit and Hare’s meta-ethnography [39], Ragin’s qualitative comparative method [45], Miles and Huberman’s methods [36], Strauss and Corbin’s constant comparison method [8], and Cohen’s post hoc power calculations [6]....
[...]