Objective measurement of test quality is one of the key issues in software testing. It has been a major research focus for the last two decades. Many test criteria have been proposed and studied for this purpose. Various kinds of rationales have been presented in support of one criterion or another. We survey the research work in this area. The notion of adequacy criteria is examined together with its role in software dynamic testing. A review of criteria classification is followed by a summary of the methods for comparison and assessment of criteria.

/pdf/software-unit-test-coverage-and-adequacy-43qk91tobg.pdf

Software unit test coverage and adequacy

Several structural testing strategies are compared in terms of their relative coverage of the program's structure and also in terms of the number of test cases needed to satisfy each strategy. Some of the deficiencies of such comparisons are discussed. >

/pdf/a-comparison-of-some-structural-testing-strategies-4bpguaazsi.pdf

A comparison of some structural testing strategies

This paper compares the fault-detecting ability of several software test data adequacy criteria. It has previously been shown that if C/sub 1/ properly covers C/sub 2/, then C/sub 1/ is guaranteed to be better at detecting faults than C/sub 2/, in the following sense: a test suite selected by independent random selection of one test case from each subdomain induced by C/sub 1/ is at least as likely to detect a fault as a test suite similarly selected using C/sub 2/. In contrast, if C/sub 1/ subsumes but does not properly cover C/sub 2/, this is not necessarily the case. These results are used to compare a number of criteria, including several that have been proposed as stronger alternatives to branch testing. We compare the relative fault-detecting ability of data flow testing, mutation testing, and the condition-coverage techniques, to branch testing, showing that most of the criteria examined are guaranteed to be better than branch testing according to two probabilistic measures. We also show that there are criteria that can sometimes be poorer at detecting faults than substantially less expensive criteria. >

Provable improvements on branch testing

ii ACKNOWLEDGMENTS First, I would like to thank my advisor, Professor Aditya P. Mathur, who gave me the impetus to look into the problem of software testing and helped me explore many of the ideas presented in this dissertation. I am also grateful to Professors for their valuable criticism and indispensable advice on this work. I thank Professor Tom Kuczek and Teresa Ruesgen for their assistance on statistical analysis. I express special thanks to Dr. Bob Horgan and Saul London at Bellcore (Bell Communications Research) who collaborated with me in part of my experiments and provided many expert suggestions, encouragement, and friendship. I am also thankful to those at Purdue and Bellcore with whom I have had many fruitful discussions while working on this research. I thank Bellcore for allowing me to use ATAC, a data ow coverage measure-Mothra, a mutation testing tool. Without these two tools, I would not have been able to conduct the experiments reported in this dissertation. Finally, my sincere thanks go to my parents Shie-Chih and Yung-Tsung Wong, my wife Nancy, our baby Rachel, and other family members for their love, understanding and support. 3.2 A mutation but not c-use, p-use, and all-uses adequate test set of P 2 : 29 3.3 A c-use, p-use, and all-uses but not mutation adequate test set of P 2 : 31 4.1 Input domain for random test case 4.2 All-uses (mutants) not covered (distinguished) by a randomly generated 4.3 Statistics on test cases generated manually to obtain all-uses and mutation adequate test sets 4.6 Average all-uses scores (%) using mutation adequate test sets obtained in Mutation 4.7 Average all-uses scores (%) using mutation adequate test sets obtained in Mutation viii Table Page 4.12 Average all-uses scores using mutation adequate test sets obtained in Mutation I excluding manually generated test cases 4.13 Average all-uses scores using mutation adequate test sets obtained in Mutation II excluding manually generated test cases ix Table Page 7.10 Percentage of test sets which are less eeective due to minimization with respect to each quartile 8.4 Average block coverage (%) of test sets with xed x LIST OF FIGURES Figure Page 1.1 Strategy to conduct an eeective software testing under time pressure and budget 3.1 Structure of programs with one deenition and one use of 3.2 Structure of programs with one deenition and multiple uses of x. : : : 25 3.3 A program in …

On mutation and data flow

A person testing a program has many methods to choose from, but little solid information about how these methods compare. Where analytic comparisons do exist, their significance is often in doubt. In this paper we examine various comparisons that have been used or proposed for test data selection and adequacy criteria. We characterize them by type and identify their strengths and weaknesses. We examine useful properties of comparisons and study the relationship between analytical and probabilistic comparisons. We find that analytical comparisons provide information of limited value, and that probabilistic comparisons overcome some of these limitations.

Comparison of program testing strategies

Empirically comparing structural test coverage metrics reveals that test sets that satisfy one metric are likely to satisfy metric as we.

P.R. McMullin

Papers

Comparison of Structural Test Coverage Metrics