Supporting Controlled Experimentation with Testing Techniques: An Infrastructure and its Potential Impact
Citations
1,276 citations
Cites background or methods from "Supporting Controlled Experimentati..."
...Our survey also provides evidence to indicate that there is a preponderance of empirical work that draws upon a comparatively small set of subjects (notably those available through the SIR repository)....
[...]
...While this is certainly evidence that SIR has been of tremendous value to the research community, it also means that many regression testing techniques are being evaluated against a limited set of programs and test suites....
[...]
...By no means is this a criticism of SIR itself or work that is based on subjects from SIR; rather, the dependency on SIR shows how time consuming and difficult it is to collect multiple versions of program source code, their test suites and associated fault information, let alone to make it publicly available....
[...]
...Tables also contain information about the origins of the studied SUT, which are classified as follows: • Siemens suite [41]: All or part of the following set of C programs—printtokens, printtokens2, schedule, schedule2, replace, tcas, totinfo, available from SIR [179]....
[...]
...• Java programs in SIR: All or part of the following set of Java programs—siena, ant, jmeter, jtopas, xml-security, nanoxml, available from SIR....
[...]
977 citations
Cites background from "Supporting Controlled Experimentati..."
...In contrast to SIR, Defects4J provides 357 real bugs for 5 large real-world programs ranging between 22,000 and 96,000 LOC....
[...]
...SIR currently provides 81 subjects written in Java, C, C++, and C#, but most of the faults are hand-seeded or obtained from mutation....
[...]
...Existing databases or repositories that provide faulty program versions [2, 4] offer only a very limited quantity of real bugs and as a consequence mutants or handseeded faults are commonly used as a substitute....
[...]
...The software-artifact infrastructure repository (SIR) [2] can be considered the first attempt to provide a database of real bugs to enable reproducibility in software testing research....
[...]
...Besides, none of the program versions that provide real bugs is accompanied by any tests, and SIR does not provide a uniform build system interface....
[...]
834 citations
Cites background from "Supporting Controlled Experimentati..."
...This is obviously true for testing as well, in which controlled experimentation is an indispensable research methodology [26]....
[...]
...Awareness of the need to unite forces is spreading, and some efforts are already being taken toward building shared data repositories, as in [26], or distributed experi-...
[...]
690 citations
Cites methods from "Supporting Controlled Experimentati..."
...The programs and test suites were from an infrastructure [4] that is designed to support controlled experimentation with software testing and regression testing techniques....
[...]
...[4] was used to obtain test suites in the following manner: In order to produce small test suites, a test case is selected at random from the test pool and is added to the suite only if it adds to the cumulative branch coverage....
[...]
...same infrastructure [4] as the programs....
[...]
573 citations
Cites methods from "Supporting Controlled Experimentati..."
...A test suite for checking the correctness of the program is presented in Table I....
[...]
References
78,012 citations
"Supporting Controlled Experimentati..." refers background in this paper
...Case studies sacrifice control, and thus, internal validity, but can include a richer context [ 51 ]....
[...]
[...]
16,383 citations
4,659 citations
"Supporting Controlled Experimentati..." refers background in this paper
...Controlled experiments focus on rigorous control of variables in an attempt to preserve internal validity and support conclusions about causality, but the limitations that result from exerting control can limit the ability to generalize results [44]....
[...]
2,947 citations
"Supporting Controlled Experimentati..." refers background or methods in this paper
...When performing a replication, researchers duplicate the experimental design of an experiment on a different sample to increase the confidence in the findings [47] or on an extended hypothesis to evaluate additional variables [5]....
[...]
...’s criteria [47], and focused primarily on whether the study involved manipulating factors to answer research questions....
[...]