Objective measurement of test quality is one of the key issues in software testing. It has been a major research focus for the last two decades. Many test criteria have been proposed and studied for this purpose. Various kinds of rationales have been presented in support of one criterion or another. We survey the research work in this area. The notion of adequacy criteria is examined together with its role in software dynamic testing. A review of criteria classification is followed by a summary of the methods for comparison and assessment of criteria.

/pdf/software-unit-test-coverage-and-adequacy-43qk91tobg.pdf

Software unit test coverage and adequacy

Extensively class tested, this text takes an innovative approach to explaining the process of software testing: it defines testing as the process of applying a few well-defined, general-purpose test criteria to a structure or model of the software. The structure of the text directly reflects the pedagogical approach and incorporates the latest innovations in testing, including techniques to test modern types of software such as OO, web applications, and embedded software.

Introduction to Software Testing: List of Figures

https://homes.cs.washington.edu/~mernst/pubs/daikon-tool-scp2007.pdf

The Daikon system for dynamic detection of likely invariants

A number of analytical models have been proposed during the past 15 years for assessing the reliability of a software system. In this paper we present an overview of the key modeling approaches, provide a critical analysis of the underlying assumptions, and assess the limitations and applicability of these models during the software development cycle. We also propose a step-by-step procedure for fitting a model and illustrate it via an analysis of failure data from a medium-sized real-time command and control software system.

Software Reliability Models: Assumptions, Limitations, and Applicability

Orthogonal defect classification (ODC), a concept that enables in-process feedback to software developers by extracting signatures on the development process from defects, is described. The ideas are evolved from an earlier finding that demonstrates the use of semantic information from defects to extract cause-effect relationships in the development process. This finding is leveraged to develop a systematic framework for building measurement and analysis methods. The authors define ODC and discuss the necessary and sufficient conditions required to provide feedback to a developer; illustrate the use of the defect type distribution to measure the progress of a product through a process; illustrate the use of the defect trigger distribution to evaluate the effectiveness and eventually the completeness of verification processes such as inspection or testing; provides sample results from pilot projects using ODC; and open the doors to a wide variety of analysis techniques for providing effective and fast feedback based on the concepts of ODC. >

Orthogonal defect classification-a concept for in-process measurements

This paper investigates a stochastic model for a software error detection process in which the growth curve of the number of detected software errors for the observed data is S-shaped. The software error detection model is a nonhomogeneous Poisson process where the mean-value function has an S-shaped growth curve. The model is applied to actual software error data. Statistical inference on the unknown parameters is discussed. The model fits the observed data better than other models.

S-Shaped Reliability Growth Modeling for Software Error Detection

This paper discusses improvements to conventional software reliability analysis models by making the assumptions on which they are based more realistic. In an actual project environment, sometimes no more information is available than reliability data obtained from a test report. The models described here are designed to resolve the problems caused by this constraint on the availability of reliability data. By utilizing the technical knowledge about a program, a test, and test data, we can select an appropriate software reliability analysis model for accurate quality assessment. The delayed S-shaped growth model, the inflection S-shaped model, and the hyperexponential model are proposed.

Software reliability analysis models

The s-shaped growth curves of detected software errors can be observed in software testing. The delayed s-shaped and inflection s-shaped software reliability growth models based on a nonhomogeneous Poisson process are discussed. The software reliability growth types of the models are investigated in terms of the error detection rate per error. In addition, a viable method for the software quality assessment, which integrates the capture-recapture method and the models above, is discussed, and its application to actual test data is illustrated.

s-Shaped Software Reliability Growth Models and Their Applications

This paper discusses improvements to conventional software reliability analysis models by making the assumptions on which they are based more realistic. That is, for example, the mutual dependency of errors in a program.

Inflection S-Shaped Software Reliability Growth Model

The authors discuss the issues of test coverage measurement in industry and justify the benefits of the measurement using a framework developed by them. Some results of experiences using test coverage measurement are outlined. The function test of large-scale system software is defined and analyzed. Based on the discussions of function test, a framework for analyzing the function test error removal process is developed. An experience-based error removal model and a cost model are proven to be useful tools for justifying test coverage measurement during function test. Data obtained from a real project are analyzed using the framework for validation. >

Mitsuru Ohba

Papers

S-Shaped Reliability Growth Modeling for Software Error Detection

Software reliability analysis models

s-Shaped Software Reliability Growth Models and Their Applications

Inflection S-Shaped Software Reliability Growth Model

Coverage measurement experience during function test