Case study is a suitable research methodology for software engineering research since it studies contemporary phenomena in its natural context. However, the understanding of what constitutes a case study varies, and hence the quality of the resulting studies. This paper aims at providing an introduction to case study methodology and guidelines for researchers conducting case studies and readers studying reports of such studies. The content is based on the authors' own experience from conducting and reading case studies. The terminology and guidelines are compiled from different methodology handbooks in other research domains, in particular social science and information systems, and adapted to the needs in software engineering. We present recommended practices for software engineering case studies as well as empirically derived and evaluated checklists for researchers and readers of case study research.

/pdf/guidelines-for-conducting-and-reporting-case-study-research-q10xdzyou5.pdf

Guidelines for conducting and reporting case study research in software engineering

Like other sciences and engineering disciplines, software engineering requires a cycle of model building, experimentation, and learning. Experiments are valuable tools for all software engineers who are involved in evaluating and choosing between different methods, techniques, languages and tools. The purpose of Experimentation in Software Engineering is to introduce students, teachers, researchers, and practitioners to empirical studies in software engineering, using controlled experiments. The introduction to experimentation is provided through a process perspective, and the focus is on the steps that we have to go through to perform an experiment. The book is divided into three parts. The first part provides a background of theories and methods used in experimentation. Part II then devotes one chapter to each of the five experiment steps: scoping, planning, execution, analysis, and result presentation. Part III completes the presentation with two examples. Assignments and statistical material are provided in appendixes. Overall the book provides indispensable information regarding empirical studies in particular for experiments, but also for case studies, systematic literature reviews, and surveys. It is a revision of the authors book, which was published in 2000. In addition, substantial new material, e.g. concerning systematic literature reviews and case study research, is introduced. The book is self-contained and it is suitable as a course book in undergraduate or graduate studies where the need for empirical studies in software engineering is stressed. Exercises and assignments are included to combine the more theoretical material with practical aspects. Researchers will also benefit from the book, learning more about how to conduct empirical studies, and likewise practitioners may use it as a cookbook when evaluating new methods or techniques before implementing them in their organization.

Experimentation in Software Engineering

Selecting a research method for empirical software engineering research is problematic because the benefits and challenges to using each method are not yet well catalogued. Therefore, this chapter describes a number of empirical methods available. It examines the goals of each and analyzes the types of questions each best addresses. Theoretical stances behind the methods, practical considerations in the application of the methods and data collection are also briefly reviewed. Taken together, this information provides a suitable basis for both understand- ing and selecting from the variety of methods applicable to empirical software engineering.

/pdf/selecting-empirical-methods-for-software-engineering-3shrao3i9n.pdf

Selecting Empirical Methods for Software Engineering Research

Context: In a previous study, we reported on a systematic literature review (SLR), based on a manual search of 13 journals and conferences undertaken in the period 1st January 2004 to 30th June 2007. Objective: The aim of this on-going research is to provide an annotated catalogue of SLRs available to software engineering researchers and practitioners. This study updates our previous study using a broad automated search. Method: We performed a broad automated search to find SLRs published in the time period 1st January 2004 to 30th June 2008. We contrast the number, quality and source of these SLRs with SLRs found in the original study. Results: Our broad search found an additional 35 SLRs corresponding to 33 unique studies. Of these papers, 17 appeared relevant to the undergraduate educational curriculum and 12 appeared of possible interest to practitioners. The number of SLRs being published is increasing. The quality of papers in conferences and workshops has improved as more researchers use SLR guidelines. Conclusion: SLRs appear to have gone past the stage of being used solely by innovators but cannot yet be considered a main stream software engineering research methodology. They are addressing a wide range of topics but still have limitations, such as often failing to assess primary study quality.

https://romisatriawahono.net/wp-content/uploads/2013/01/Kitchenham-Systematic-literature-reviews-in-software-engineering-%E2%80%93-A-tertiary-study-2012.pdf

Systematic literature reviews in software engineering - A tertiary study

Scientists spend an increasing amount of time building and using software. However, most scientists are never taught how to do this efficiently. As a result, many are unaware of tools and practices that would allow them to write more reliable and maintainable code with less effort. We describe a set of best practices for scientific software development that have solid foundations in research and experience, and that improve scientists' productivity and the reliability of their software.

Software is as important to modern scientific research as telescopes and test tubes. From groups that work exclusively on computational problems, to traditional laboratory and field scientists, more and more of the daily operation of science revolves around developing new algorithms, managing and analyzing the large amounts of data that are generated in single research projects, combining disparate datasets to assess synthetic problems, and other computational tasks.

Scientists typically develop their own software for these purposes because doing so requires substantial domain-specific knowledge. As a result, recent studies have found that scientists typically spend 30% or more of their time developing software [1],[2]. However, 90% or more of them are primarily self-taught [1],[2], and therefore lack exposure to basic software development practices such as writing maintainable code, using version control and issue trackers, code reviews, unit testing, and task automation.

We believe that software is just another kind of experimental apparatus [3] and should be built, checked, and used as carefully as any physical apparatus. However, while most scientists are careful to validate their laboratory and field equipment, most do not know how reliable their software is [4],[5]. This can lead to serious errors impacting the central conclusions of published research [6]: recent high-profile retractions, technical comments, and corrections because of errors in computational methods include papers in Science [7],[8], PNAS [9], the Journal of Molecular Biology [10], Ecology Letters [11],[12], the Journal of Mammalogy [13], Journal of the American College of Cardiology [14], Hypertension [15], and The American Economic Review [16].

In addition, because software is often used for more than a single project, and is often reused by other scientists, computing errors can have disproportionate impacts on the scientific process. This type of cascading impact caused several prominent retractions when an error from another group's code was not discovered until after publication [6]. As with bench experiments, not everything must be done to the most exacting standards; however, scientists need to be aware of best practices both to improve their own approaches and for reviewing computational work by others.

This paper describes a set of practices that are easy to adopt and have proven effective in many research settings. Our recommendations are based on several decades of collective experience both building scientific software and teaching computing to scientists [17],[18], reports from many other groups [19]–, guidelines for commercial and open source software development [26],, and on empirical studies of scientific computing [28]–[31] and software development in general (summarized in [32]). None of these practices will guarantee efficient, error-free software development, but used in concert they will reduce the number of errors in scientific software, make it easier to reuse, and save the authors of the software time and effort that can used for focusing on the underlying scientific questions.

Our practices are summarized in Box 1; labels in the main text such as “(1a)” refer to items in that summary. For reasons of space, we do not discuss the equally important (but independent) issues of reproducible research, publication and citation of code and data, and open science. We do believe, however, that all of these will be much easier to implement if scientists have the skills we describe.


Box 1. Summary of Best Practices

Write programs for people, not computers.


A program should not require its readers to hold more than a handful of facts in memory at once.


Make names consistent, distinctive, and meaningful.


Make code style and formatting consistent.




Let the computer do the work.


Make the computer repeat tasks.


Save recent commands in a file for re-use.


Use a build tool to automate workflows.




Make incremental changes.


Work in small steps with frequent feedback and course correction.


Use a version control system.


Put everything that has been created manually in version control.




Don't repeat yourself (or others).


Every piece of data must have a single authoritative representation in the system.


Modularize code rather than copying and pasting.


Re-use code instead of rewriting it.




Plan for mistakes.


Add assertions to programs to check their operation.


Use an off-the-shelf unit testing library.


Turn bugs into test cases.


Use a symbolic debugger.




Optimize software only after it works correctly.


Use a profiler to identify bottlenecks.


Write code in the highest-level language possible.




Document design and purpose, not mechanics.


Document interfaces and reasons, not implementations.


Refactor code in preference to explaining how it works.


Embed the documentation for a piece of software in that software.




Collaborate.


Use pre-merge code reviews.


Use pair programming when bringing someone new up to speed and when tackling particularly tricky problems.


Use an issue tracking tool.








Write Programs for People, Not Computers
Scientists writing software need to write code that both executes correctly and can be easily read and understood by other programmers (especially the author's future self). If software cannot be easily read and understood, it is much more difficult to know that it is actually doing what it is intended to do. To be productive, software developers must therefore take several aspects of human cognition into account: in particular, that human working memory is limited, human pattern matching abilities are finely tuned, and human attention span is short [33]–[37].

First, a program should not require its readers to hold more than a handful of facts in memory at once (1a). Human working memory can hold only a handful of items at a time, where each item is either a single fact or a “chunk” aggregating several facts [33],[34], so programs should limit the total number of items to be remembered to accomplish a task. The primary way to accomplish this is to break programs up into easily understood functions, each of which conducts a single, easily understood, task. This serves to make each piece of the program easier to understand in the same way that breaking up a scientific paper using sections and paragraphs makes it easier to read.

Second, scientists should make names consistent, distinctive, and meaningful (1b). For example, using non-descriptive names, like a and foo, or names that are very similar, like results and results2, is likely to cause confusion.

Third, scientists should make code style and formatting consistent (1c). If different parts of a scientific paper used different formatting and capitalization, it would make that paper more difficult to read. Likewise, if different parts of a program are indented differently, or if programmers mix CamelCaseNaming and pothole_case_naming, code takes longer to read and readers make more mistakes [35],[36].

/pdf/best-practices-for-scientific-computing-fj5m4m299p.pdf

Best Practices for Scientific Computing

Over the past decade we have performed a sustained series of qualitative studies of software development practice, focusing on social factors. Using an ethnographically-informed approach, we have addressed four areas of software practice: software quality management systems, the emergence of object technology, professional end user development and agile development. Several issues have arisen from this experience, including the nature of research questions that such studies can address, the advantages and challenges associated with being a member of the community under study, and how to maintain rigour in data collection. In this paper, we will draw on our studies to illustrate our approach and to discuss these and other issues.

Ethnographically-informed empirical studies of software practice

This paper describes a case study of software engineers developing a library of software components for a group of research scientists, using a traditional, staged, document-led methodology. The case study reveals two problems with the use of the methodology. The first is that it demands an upfront articulation of requirements, whereas the scientists had experience, and hence expectations, of emergent requirements; the second is that the project documentation does not suffice to construct a shared understanding. Reflecting on our case study, we discuss whether combining agile elements with a traditional methodology might have alleviated these problems. We then argue that the rich picture painted by the case study, and the reflections on methodology that it inspires, has a relevance that reaches beyond the original context of the study.

When Software Engineers Met Research Scientists: A Case Study

Not all scientific computing is high-performance computing—the variety of scientific software is huge. Such software might be complex simulation software developed and running on a high-performance computer, or software developed on a PC for embedding into instruments; for manipulating, analyzing or visualizing data or for orchestrating workflows. This special issue provides some flavor of that variety. It also explores the question of how the development of scientific software can be improved.

/pdf/developing-scientific-software-2virvnbsbz.pdf

Developing Scientific Software

By the term 'professional end user developers' we mean people such as research scientists who work in highly technical, knowledge-rich domains and who develop software in order to further their professional goals. In common with other end user developers, professional end user developers do not describe themselves as software engineers and have no formal training in software engineering. They differ from most other end user developers, however, in that learning programming languages rarely presents them with any problem. In this paper, drawing on data from field studies of different groups of professional end users, we examine the problems that such people face in meeting the demands of software development given the culture in which they work and their normal development practice. Understanding these problems is an essential prerequisite to developing tools, techniques etcetera to support professional end user development.

/pdf/some-problems-of-professional-end-user-developers-2l0kzqywqu.pdf

Some Problems of Professional End User Developers

Much of the knowledge used within an XP team is tacit, i.e. it is hidden and intangible. Two tangible artefacts that carry information about the team's work are the index cards which capture stories and tasks to be implemented and the wall where they are displayed (which we refer to as the 'Wall'). It is widely acknowledged that these are key elements supporting the work of the XP team, but no systematic investigation of their role has been reported to date. In this paper, we focus on the use of these artefacts within one XP team. We use distributed cognition, a framework for analysing collaborative work, to explicate the information flows in, around and within the team that are supported by the index cards and the Wall. We then interrogate the models produced using this analysis to answer 'what if' questions.

/pdf/the-role-of-story-cards-and-the-wall-in-xp-teams-a-2eodaqfdye.pdf

Judith Segal

Papers

Ethnographically-informed empirical studies of software practice

When Software Engineers Met Research Scientists: A Case Study

Developing Scientific Software

Some Problems of Professional End User Developers

The role of story cards and the wall in XP teams: a distributed cognition perspective