Case study is a suitable research methodology for software engineering research since it studies contemporary phenomena in its natural context. However, the understanding of what constitutes a case study varies, and hence the quality of the resulting studies. This paper aims at providing an introduction to case study methodology and guidelines for researchers conducting case studies and readers studying reports of such studies. The content is based on the authors' own experience from conducting and reading case studies. The terminology and guidelines are compiled from different methodology handbooks in other research domains, in particular social science and information systems, and adapted to the needs in software engineering. We present recommended practices for software engineering case studies as well as empirically derived and evaluated checklists for researchers and readers of case study research.

/pdf/guidelines-for-conducting-and-reporting-case-study-research-q10xdzyou5.pdf

Guidelines for conducting and reporting case study research in software engineering

Like other sciences and engineering disciplines, software engineering requires a cycle of model building, experimentation, and learning. Experiments are valuable tools for all software engineers who are involved in evaluating and choosing between different methods, techniques, languages and tools. The purpose of Experimentation in Software Engineering is to introduce students, teachers, researchers, and practitioners to empirical studies in software engineering, using controlled experiments. The introduction to experimentation is provided through a process perspective, and the focus is on the steps that we have to go through to perform an experiment. The book is divided into three parts. The first part provides a background of theories and methods used in experimentation. Part II then devotes one chapter to each of the five experiment steps: scoping, planning, execution, analysis, and result presentation. Part III completes the presentation with two examples. Assignments and statistical material are provided in appendixes. Overall the book provides indispensable information regarding empirical studies in particular for experiments, but also for case studies, systematic literature reviews, and surveys. It is a revision of the authors book, which was published in 2000. In addition, substantial new material, e.g. concerning systematic literature reviews and case study research, is introduced. The book is self-contained and it is suitable as a course book in undergraduate or graduate studies where the need for empirical studies in software engineering is stressed. Exercises and assignments are included to combine the more theoretical material with practical aspects. Researchers will also benefit from the book, learning more about how to conduct empirical studies, and likewise practitioners may use it as a cookbook when evaluating new methods or techniques before implementing them in their organization.

Experimentation in Software Engineering

Where the creation, understanding, and assessment of software testing and regression testing techniques are concerned, controlled experimentation is an indispensable research methodology. Obtaining the infrastructure necessary to support such experimentation, however, is difficult and expensive. As a result, progress in experimentation with testing techniques has been slow, and empirical data on the costs and effectiveness of techniques remains relatively scarce. To help address this problem, we have been designing and constructing infrastructure to support controlled experimentation with testing and regression testing techniques. This paper reports on the challenges faced by researchers experimenting with testing techniques, including those that inform the design of our infrastructure. The paper then describes the infrastructure that we are creating in response to these challenges, and that we are now making available to other researchers, and discusses the impact that this infrastructure has had and can be expected to have.

/pdf/supporting-controlled-experimentation-with-testing-pe7sdxpo8h.pdf

Supporting Controlled Experimentation with Testing Techniques: An Infrastructure and its Potential Impact

The empirical assessment of test techniques plays an important role in software testing research. One common practice is to instrument faults, either manually or by using mutation operators. The latter allows the systematic, repeatable seeding of large numbers of faults; however, we do not know whether empirical results obtained this way lead to valid, representative conclusions. This paper investigates this important question based on a number of programs with comprehensive pools of test cases and known faults. It is concluded that, based on the data available thus far, the use of mutation operators is yielding trustworthy results (generated mutants are similar to real faults). Mutants appear however to be different from hand-seeded faults that seem to be harder to detect than real faults.

/pdf/is-mutation-an-appropriate-tool-for-testing-experiments-2bbva6pfdl.pdf

Is mutation an appropriate tool for testing experiments

We present a comprehensive suite of experimentation on the subject of learning from imbalanced data. When classes are imbalanced, many learning algorithms can suffer from the perspective of reduced performance. Can data sampling be used to improve the performance of learners built from imbalanced data? Is the effectiveness of sampling related to the type of learner? Do the results change if the objective is to optimize different performance metrics? We address these and other issues in this work, showing that sampling in many cases will improve classifier performance.

/pdf/experimental-perspectives-on-learning-from-imbalanced-data-29h0tmgbxs.pdf

Experimental perspectives on learning from imbalanced data

The purpose of Experimentation in Software Engineering: An Introduction is to introduce students, teachers, researchers, and practitioners to experimentation and experimental evaluation with a focus on software engineering. The objective is, in particular, to provide guidelines for performing experiments evaluating methods, techniques and tools in software engineering. The introduction is provided through a process perspective. The focus is on the steps that we go through to perform experiments and quasi-experiments. The process also includes other types of empirical studies. The motivation for the book emerged from the need for support we experienced when turning our software engineering research more experimental. Several books are available which either treat the subject in very general terms or focus on some specific part of experimentation; most focus on the statistical methods in experimentation. These are important, but there were few books elaborating on experimentation from a process perspective, none addressing experimentation in software engineering in particular. The scope of Experimentation in Software Engineering: An Introduction is primarily experiments in software engineering as a means for evaluating methods, techniques and tools. The book provides some information regarding empirical studies in general, including both case studies and surveys. The intention is to provide a brief understanding of these strategies and in particular to relate them to experimentation. Experimentation in Software Engineering: An Introduction is suitable for use as a textbook or a secondary text for graduate courses, and for researchers and practitioners interested in an empirical approach to software engineering. (Less)

Experimentation in Software Engineering: An Introduction

The paper presents the idea of usage oriented requirements engineering, an extension of use case driven analysis. The main objective is to achieve a requirements engineering process resulting in a model which captures both functional requirements and system usage aspects in a comprehensive manner. The paper presents the basic concepts and the process of usage oriented requirements engineering, and the synthesized usage model resulting from this process. The role of this model in system development, and its potential applications are also discussed.

Improving the use case driven approach to requirements engineering

The Personal Software Process (PSP) has during the last couple of years gained attention as a way to individual improvements in software development. The PSP is introduced to students and engineers through a course, which introduces a personal software development process. The personal software development process is improved in steps during the course and a collection of methods is introduced to support the personal development process. The question is, however, how do these methods influence the performance of an individual engineer? This question has been studied in a study made at the Software Engineering Institute, and the study has shown that the methods in the PSP have a positive effect on the performance of the individuals. There is however a need to replicate this study to confirm the findings in other settings and with other individuals.

This paper describes a replication of the study made at the Software Engineering Institute. Both the original study and this replication are made on data reported from the students taking the PSP course. The differences between the two studies are the programming languages used, which held the courses, the class sizes, and the experiences of the students. In summary, the results from this replication confirm the results in the original study: Size estimation accuracy gets better, the defect density gets lower, the defects are found earlier and that the pre-compile yield gets better during the PSP course. Basically, the two studies show that the methods in the PSP help engineers to improve their performance.

A Replicated Empirical Study of the Impact of the Methodsin the PSP on Individual Engineers

Software reliability is an important characteristic for most systems, but due to its dynamic properties it is hard to assess until very late in the development. Nevertheless, the testing must be planned to meet the reliability requirements. In test planning, the notion of usage coverage may be used as an indicator of reliability as they are correlated with each other. When the testing is planned, the test cases to be run and the usage profile are derived. The usage profile is estimated using the available information of the expected usage. There are therefore some uncertainties in the estimated usage profile. The paper presents and evaluates a method for analysing the impact of uncertainties in the usage profile on the usage coverage. The method is applied during test planning to evaluate how sensitive the usage coverage is to usage profile uncertainties. Different usage profiles are simulated using the expected usage profile and an uncertainty expressed as a percentage value or a range within which the changed usage profile may vary. Test cases are derived from the expected usage profile and the resulting usage coverage is estimated for each simulated usage profile. Thus the impact in the usage coverage can be analysed. The presented analysis method is illustrated with an example system. One of the conclusions from this first study of the method is that the sparseness of the usage profile, which is defined by limitations in the usage pattern, reduces the impact of the usage profile uncertainties. Further, the number of test cases is an important factor in the sensitivity to uncertainties in the usage profile.

Assessing the sensitivity to usage profile changes in test planning

Statistical Usage Testing is a scientific approach to software testing. Instead of ad hoc selection of test cases, the selection is controlled by a usage specification which models the anticipated software usage and hence not the system itself. The test cases are derived from the usage specification and constitute a sample from the operational usage. Thus the software reliability can be certified by applying a statistical model.

Anders Wesslén

Papers

Experimentation in Software Engineering: An Introduction

Improving the use case driven approach to requirements engineering

A Replicated Empirical Study of the Impact of the Methodsin the PSP on Individual Engineers

Assessing the sensitivity to usage profile changes in test planning

Statistical Usage Testing Using SDL