Objective measurement of test quality is one of the key issues in software testing. It has been a major research focus for the last two decades. Many test criteria have been proposed and studied for this purpose. Various kinds of rationales have been presented in support of one criterion or another. We survey the research work in this area. The notion of adequacy criteria is examined together with its role in software dynamic testing. A review of criteria classification is followed by a summary of the methods for comparison and assessment of criteria.

/pdf/software-unit-test-coverage-and-adequacy-43qk91tobg.pdf

Software unit test coverage and adequacy

Model-based testing is a new and evolving technique for generating a suite of test cases from requirements. Testers using this approach concentrate on a data model and generation infrastructure instead of hand-crafting individual tests. Several relatively small studies have demonstrated how combinatorial test generation techniques allow testers to achieve broad coverage of the input domain with a small number of tests. We have conducted several relatively large projects in which we applied these techniques to systems with millions of lines of code. Given the complexity of testing, the model-based testing approach was used in conjunction with test automation harnesses. Since no large empirical study has been conducted to measure efficacy of this new approach, we report on our experience with developing tools and methods in support of model-based testing. The four case studies presented here offer details and results of applying combinatorial test-generation techniques on a large scale to diverse applications. Based on the four projects, we offer our insights into what works in practice and our thoughts about obstacles to transferring this technology into testing organizations.

Model-based testing in practice

Architecture-based approach to reliability assessment of software systems

Statistical testing of software establishes a basis for statistical inference about a software system's expected field quality. This paper describes a method for statistical testing based on a Markov chain model of software usage. The significance of the Markov chain is twofold. First, it allows test input sequences to be generated from multiple probability distributions, making it more general than many existing techniques. Analytical results associated with Markov chains facilitate informative analysis of the sequences before they are generated, indicating how the test is likely to unfold. Second, the test input sequences generated from the chain and applied to the software are themselves a stochastic model and are used to create a second Markov chain to encapsulate the history of the test, including any observed failure information. The influence of the failures is assessed through analytical computations on this chain. We also derive a stopping criterion for the testing process based on a comparison of the sequence generating properties of the two chains. >

/pdf/a-markov-chain-model-for-statistical-software-testing-26830n6gtt.pdf

A Markov chain model for statistical software testing

Where do most vulnerabilities occur in software? Our Vulture tool automatically mines existing vulnerability databases and version archives to map past vulnerabilities to components. The resulting ranking of the most vulnerable components is a perfect base for further investigations on what makes components vulnerable.In an investigation of the Mozilla vulnerability history, we surprisingly found that components that had a single vulnerability in the past were generally not likely to have further vulnerabilities. However, components that had similar imports or function calls were likely to be vulnerable.Based on this observation, we were able to extend Vulture by a simple predictor that correctly predicts about half of all vulnerable components, and about two thirds of all predictions are correct. This allows developers and project managers to focus their their efforts where it is needed most: "We should look at nsXPInstallManager because it is likely to contain yet unknown vulnerabilities.".

/pdf/predicting-vulnerable-software-components-1kxxqnxmv7.pdf

Predicting vulnerable software components

Formulas for estimating the probability of failure when testing reveals no errors are introduced. These formulas incorporate random testing results, information about the input distribution; and prior assumptions about the probability of failure of the software. The formulas are not restricted to equally likely input distributions, and the probability of failure estimate can be adjusted when assumptions about the input distribution change. The formulas are based on a discrete sample space statistical model of software and include Bayesian prior assumptions. Reusable software and software in life-critical applications are particularly appropriate candidates for this type of analysis. >

Estimating the probability of failure when testing reveals no failures

Error flow analysis is the study of how errors originate, spread, and propagate during program execution based on the three steps of the fault/failure model: execution, infection, and propagation. These three steps are defined relative to a virtual computer-by judiciously selecting the instruction set and data state of this computer the need for infection analysis can be reduced or eliminated in favor of execution and propagation analysis. A key aspect of propagation analysis is injecting errors into the data state and tracing their expect. Perturbation analysis injects errors by directly modifying the data state of on executing program. The resulting code that is executed-the tail code-is analyzed for its error-flow behavior. Perturbation analysis is a language-independent and efficient method of characterizing the propagation rate of each tail function, the function computed by all tail code originating at a given location. This paper defines a model for perturbation analysis, and uses the model to explain the performance of analysis techniques (e.g. statement, data flow, and mutation analysis).

Perturbation analysis of computer programs

Error flow analysis and testing techniques focus on the introduction of errors through code faults into data states of an executing program, and their subsequent cancellation or propagation to output. The goals and limitations of several error flow techniques are discussed, including mutation analysis, fault-based testing, PIE analysis, and dynamic impact analysis. The attributes desired of a good error flow technique are proposed, and a model called dynamic error flow analysis (DEFA) is described that embodies many of these attributes. A testing strategy is proposed that uses DEFA information to select an optimal set of test paths and to quantify the results of successful testing. An experiment is presented that illustrates this testing strategy. In this experiment, the proposed testing strategy outperforms mutation testing in catching arbitrary data state errors.

An empirical, path-oriented approach to software analysis and testing

Perturbation analysis involves changing the data state at a location in an executing program and determining the effect of that change on the program's remaining computation. This technique has been used to estimate propagation rates during fault sensitivity analysis, and also to compare test sets and testing strategies. This paper describes an error flow model that allows emphasis to be placed on propagation analysis, which is implemented using data state perturbation. A perturbation-adequate testing strategy is defined that is similar in concept to mutation testing, and some results are presented that support the coupling effect and the consistency of perturbation-adequate testing. Some observations are made that relate mutation to perturbation.

Branson W. Murrill

Papers

Estimating the probability of failure when testing reveals no failures

Perturbation analysis of computer programs

An empirical, path-oriented approach to software analysis and testing

An empirical, path-oriented approach to software analysis and testing

A perturbation-based testing strategy