Empirically based theories are generally perceived as foundational to science. However, in many disciplines, the nature, role and even the necessity of theories remain matters for debate, particularly in young or practical disciplines such as software engineering. This article reports a systematic review of the explicit use of theory in a comprehensive set of 103 articles reporting experiments, from of a total of 5,453 articles published in major software engineering journals and conferences in the decade 1993-2002. Of the 103 articles, 24 use a total of 40 theories in various ways to explain the cause-effect relationship(s) under investigation. The majority of these use theory in the experimental design to justify research questions and hypotheses, some use theory to provide post hoc explanations of their results, and a few test or modify theory. A third of the theories are proposed by authors of the reviewed articles. The interdisciplinary nature of the theories used is greater than that of research in software engineering in general. We found that theory use and awareness of theoretical issues are present, but that theory-driven research is, as yet, not a major issue in empirical software engineering. Several articles comment explicitly on the lack of relevant theory. We call for an increased awareness of the potential benefits of involving theory, when feasible. To support software engineering researchers who wish to use theory, we show which of the reviewed articles on which topics use which theories for what purposes, as well as details of the theories' characteristics

A Systematic Review of Theory Use in Software Engineering Experiments

The objective of this thesis is to collect guidelines for the replication of software engineering experiments, and to refine and supplement these guidelines where needed. These guidelines will hopefully contribute to the establishment of an empirical framework for the software engineering discipline.

/pdf/replication-of-controlled-experiments-in-empirical-software-5f8v2rx9vv.pdf

Replication of Controlled Experiments in Empirical Software Engineering - A Survey

Software engineering processes depend on the context they are applied in. Thus, it is risky to determine the best processes for a given project context without empirical data from this context. This report summarizes important issues on the state of the art in empirical studies in order to provide researchers with the background to conduct their own empirical studies and for practitioners to classify existing empirical studies regarding their usefulness. This report gives an overview on the state of the art in empirical studies in software engineering by prescribing a high-level process for empirical studies , which is refined for the three most often used empirical strategies: Controlled experiments, case studies, and surveys. For each strategy this work discusses representative application reports. The appendix lists a glossary of empirical vocabulary, a standard report outline , a bibliography, and experimental material available on-line. 3 Controlled Experiments 19 3.1 Study Definition 21 3.1.1 Determine the goal of the study and an informal list of hypotheses 22 3.1.2 Record the goal in the measurement goal template 23 3.1.3 Determine context of the experiment 25 3.2 Experiment design 26 3.2.1 Operationalize the goal by identifying and quantifying the dependent and independent variables 26 3.2.2 Select an experimental design that satisfies the experiment purpose 27 3.2.3 Consider validity threats 31 3.2.4 Determine the necessary criteria for potential subjects 32 3.2.5 Determine options for data collection to increase the potential benefit from the experiment 33 3.3

State-of-the-Art in Empirical Studies

Background: Like any research discipline, software engineering research must be of a certain quality to be valuable. High quality research in software engineering ensures that knowledge is accumulated and helpful advice is given to the industry. One way of assessing research quality is to conduct systematic reviews of the published research literature. Objective: The purpose of this work was to assess the quality of published experiments in software engineering with respect to the validity of inference and the quality of reporting. More specifically, the aim was to investigate the level of statistical power, the analysis of effect size, the handling of selection bias in quasi-experiments, and the completeness and consistency of the reporting of information regarding subjects, experimental settings, design, analysis, and validity. Furthermore, the work aimed at providing suggestions for improvements, using the potential deficiencies detected as a basis. Method: The quality was assessed by conducting a systematic review of the 113 experiments published in nine major software engineering journals and three conference proceedings in the decade 1993-2002. Results: The review revealed that software engineering experiments were generally designed with unacceptably low power and that inadequate attention was paid to issues of statistical power. Effect sizes were sparsely reported and not interpreted with respect to their practical importance for the particular context. There seemed to be little awareness of the importance of controlling for selection bias in quasi-experiments. Moreover, the review revealed a need for more complete and standardized reporting of information, which is crucial for understanding software engineering experiments and judging their results. Implications: The consequence of low power is that the actual effects of software engineering technologies will not be detected to an acceptable extent. The lack of reporting of effect sizes and the improper interpretation of effect sizes result in ignorance of the practical importance, and thereby the relevance to industry, of experimental results. The lack of control for selection bias in quasi-experiments may make these experiments less credible than randomized experiments. This is an unsatisfactory situation, because quasiexperiments serve an important role in investigating cause-effect relationships in software engineering, for example, in industrial settings. Finally, the incomplete and unstandardized reporting makes it difficult for the reader to understand an experiment and judge its results. Conclusions: Insufficient quality was revealed in the reviewed experiments. This has implications for inferences drawn from the experiments and might in turn lead to the accumulation of erroneous information and the offering of misleading advice to the industry. Ways to improve this situation are suggested.

/pdf/quality-of-design-analysis-and-reporting-of-software-18y28ezeow.pdf

Quality of Design, Analysis and Reporting of Software Engineering Experiments:A Systematic Review

This paper reports the results of a controlled experiment undertaken to investigate whether the methodology support offered by a CASE tool does have an impact on the tool's acceptance and actual use by individuals.Subjects used the process modelling tool SPEARMINT to complete a partial process model and remove all inconsistencies. Half the subjects used a variant of SPEARMINT that corrected consistency violations automatically and silently, whilst the other half used a variant of SPEARMINT that told them about inconsistencies both immediately and persistently but without automatic correction. Measurement of acceptance and prediction of actual use was based on the technology acceptance model, supplemented by beliefs about consistency rules. The impact of form of automated consistency assurance applied or hierarchical consistency rules was found to be significant at the 0.05 level with a type I error of 0.027, explaining 71.6% of the variance in CASE tool acceptance. However, intention to use and thus predicted use was of the same size for both variants of SPEARMINT, whereas perceived usefulness and perceived ease of use were affected contrarily.Internal validity of the findings was threatened by validity and reliability issues related to beliefs about consistency rules. Here, further research is needed to develop valid constructs and reliable scales. Following the experiment, a small survey among experienced users of SPEARMINT found that different forms of automated consistency assurance were preferred depending on individual, consistency rule, and task characteristics. Based on these findings, it is recommended that vendors should provide CASE tools with adaptable methodology support, which allow their users to fit automated consistency assurance to the task at hand.

Methodology Support in CASE Tools and Its Impact on Individual Acceptance and Use: A Controlled Experiment

This paper reports the results of an experiment undertaken for the CADPRO (Constraints And the Decision PROject) project. Subjects with varied experience produced data flow diagrams (DFDs) using a DFD tool generated by CASEMaker, a meta-CASE tool. Half the subjects received routine notice of instances of internal (as opposed to hierarchical) methodological constraint violations via an unobtrusive window whilst the other half did not. The DFD tool automatically recorded subjects' delivery and constraint profiles. Video records, observer notes, and subject debriefings were also used to yield other performance data. While evidence was found in support of the research model underpinning the CADPRO project, the model needs to be revised to take into account the affects of human-computer interface constraints and the different speeds with which people work. We learnt an important lesson about subject randomisation, which is not to assume that all subjects can be treated alike if they share the minimum necessary experience thought required of the problem. We believe it is important for every subject-based experiment to consider and understand the performance of individuals. Because of the complexity of constraint environments in CASE tools we also conclude that studies comparing extreme programming approaches with conventional CASE tool approaches are needed to help determine if the struggle to understand the constraint environment at a high level of abstraction is worthwhile or not. Further experiments, possibly replication variants of this one, are needed to help validate our interpretations.

Early Lifecycle Work: Influence of Individual Characteristics,Methodological Constraints, and Interface Constraints

This paper describes an experiment that investigated the effects on design productivity and quality of displaying design constraint violations or design errors to the designer when using a case tool for early lifecycle system design. A laboratory experiment was conducted in which the design activity was carried out using an instrumented tool that was developed to support the experiment. Two versions of the case tool were used, one which displayed constraint violations and one which did not. We found that the display of constraint violations had a significant impact on the productivity of the designer by slowing the design process. There was not a statistically significant difference in the quality of designs at the end of the design exercise although subjects with constraint violation notification had 61% fewer errors in their designs.

Fredrik Utbult

Papers

Early Lifecycle Work: Influence of Individual Characteristics,Methodological Constraints, and Interface Constraints

The Effect of Constraint Notification within a Case Tool Environment of Design Productivity and Quality