Context: We are strong advocates of evidence-based software engineering (EBSE) in general and systematic literature reviews (SLRs) in particular. We believe it is essential that the SLR methodology is used constructively to support software engineering research. Objective: This study aims to assess the value of mapping studies which are a form of SLR that aims to identify and categorise the available research on a broad software engineering topic. Method: We used a multi-case, participant-observer case study using five examples of studies that were based on preceding mapping studies. We also validated our results by contacting two other researchers who had undertaken studies based on preceding mapping studies and by assessing review comments related to our follow-on studies. Results: Our original case study identified 11 unique benefits that can accrue from basing research on a preceding mapping study of which only two were case specific. We also identified nine problems associated with using preceding mapping studies of which two were case specific. These results were consistent with the information obtained from the validation activities. We did not find an example of an independent research group making use of a mapping study produced by other researchers. Conclusion: Mapping studies can save time and effort for researchers and provide baselines to assist new research efforts. However, they must be of high quality in terms of completeness and rigour if they are to be a reliable basis for follow-on research.

/pdf/using-mapping-studies-as-the-basis-for-further-research-a-30nrr2jq2u.pdf

Using mapping studies as the basis for further research - A participant-observer case study

ContextComparing and contrasting evidence from multiple studies is necessary to build knowledge and reach conclusions about the empirical support for a phenomenon. Therefore, research synthesis is at the center of the scientific enterprise in the software engineering discipline. ObjectiveThe objective of this article is to contribute to a better understanding of the challenges in synthesizing software engineering research and their implications for the progress of research and practice. MethodA tertiary study of journal articles and full proceedings papers from the inception of evidence-based software engineering was performed to assess the types and methods of research synthesis in systematic reviews in software engineering. ResultsAs many as half of the 49 reviews included in the study did not contain any synthesis. Of the studies that did contain synthesis, two thirds performed a narrative or a thematic synthesis. Only a few studies adequately demonstrated a robust, academic approach to research synthesis. ConclusionWe concluded that, despite the focus on systematic reviews, there is limited attention paid to research synthesis in software engineering. This trend needs to change and a repertoire of synthesis methods needs to be an integral part of systematic reviews to increase their significance and utility for research and practice.

/pdf/research-synthesis-in-software-engineering-40heqp30v5.pdf

Research synthesis in software engineering

BACKGROUND-Software Process Improvement (SPI) is a systematic approach to increase the efficiency and effectiveness of a software development organization and to enhance software products. OBJECTIVE-This paper aims to identify and characterize evaluation strategies and measurements used to assess the impact of different SPI initiatives. METHOD-The systematic literature review includes 148 papers published between 1991 and 2008. The selected papers were classified according to SPI initiative, applied evaluation strategies, and measurement perspectives. Potential confounding factors interfering with the evaluation of the improvement effort were assessed. RESULTS-Seven distinct evaluation strategies were identified, wherein the most common one, “Pre-Post Comparison,” was applied in 49 percent of the inspected papers. Quality was the most measured attribute (62 percent), followed by Cost (41 percent), and Schedule (18 percent). Looking at measurement perspectives, “Project” represents the majority with 66 percent. CONCLUSION-The evaluation validity of SPI initiatives is challenged by the scarce consideration of potential confounding factors, particularly given that “Pre-Post Comparison” was identified as the most common evaluation strategy, and the inaccurate descriptions of the evaluation context. Measurements to assess the short and mid-term impact of SPI initiatives prevail, whereas long-term measurements in terms of customer satisfaction and return on investment tend to be less used.

/pdf/evaluation-and-measurement-of-software-process-improvement-a-3mqqc927q5.pdf

Evaluation and Measurement of Software Process Improvement—A Systematic Literature Review

What's up with software metrics? - A preliminary mapping study

We are strong advocates of evidence-based software engineering (EBSE) in general and systematic literature reviews (SLRs) in particular. We believe it is essential that the SLR methodology is being used constructively to support software engineering research. Aim: This study aims to assess the value of mapping studies which are a form of SLR that aims to identify and categorise the available research on a specific topic. Methods: We use a multi-case, participant observer case study using five examples of studies that were based on preceding mapping studies. Results: We identified 13 unique benefits that can accrue from basing research on a preceding mapping study of which only 2 were case specific. We also identified 9 problems associated with using preceding mapping studies of which two were case specific. Conclusions: Mapping studies can save time and effort for researchers and provide baselines to assist new research efforts. However, they must be of high quality in terms of completeness and rigour if they are to be a reliable basis for follow-on research.

/pdf/the-value-of-mapping-studies-a-participantobserver-case-36omlb9haj.pdf

The value of mapping studies: a participantobserver case study

The present work provides a summary of the state of art in software measures by means of a systematic review on the current literature. Nowadays, many companies need to answer the following questions: How to measure?, When to measure and What to measure?. There have been a lot of efforts made to attempt to answer these questions, and this has resulted in a large amount of data what is sometimes confusing and unclear information. This needs to be properly processed and classified in order to provide a better overview of the current situation. We have used a Measurement Software Ontology to classify and put the amount of data in this field in order. We have also analyzed the results of the systematic review, to show the trends in the software measurement field and the software process on which the measurement efforts have focused. It has allowed us to discover what parts of the process are not supported enough by measurements, to thus motivate future research in those areas.

Oswaldo Gómez

Papers

A Systematic Review Measurement in Software Engineering: State-of-the-Art in Measures