Methodological Issues in Observational Studies

TLDR: In this paper, the authors define a methodology for applying observational studies in empirical software engineering, providing guidelines on how to conduct such studies, how to analyze the data, and how to report the studies themselves.

Abstract: Background: Starting from the 1960s, practitioners and researchers have looked for ways to empirically investigate new technologies such as inspecting the effectiveness of new methods, tools, or practices. With this purpose, the empirical software engineering domain started to identify different empirical methods, borrowing them from various domains such as medicine, biology, and psychology. Nowadays, a variety of empirical methods are commonly applied in software engineering, ranging from controlled and quasi-controlled experiments to case studies, from systematic literature reviews to the newly introduced multivocal literature reviews. However, to date, the only available method for proving any cause-effect relationship are controlled experiments. Objectives: The goal of the thesis is introducing new methodologies for studying causality in empirical software engineering. Methods: Other fields use observational studies for proving causality. They allow observing the effect of a risk factor and testing this without trying to change who is or is not exposed to it. As an example, with an observational study it is possible to observe the effect of pollution on the growth of a forest or the effect of different factors on development productivity without the need of waiting years for the forest to grow or exposing developers to a specific treatment. Conclusion: In this thesis, we aim at defining a methodology for applying observational studies in empirical software engineering, providing guidelines on how to conduct such studies, how to analyze the data, and how to report the studies themselves.