Estimating remaining defects (or failures) in software can help test managers make release decisions during testing. Several methods exist to estimate defect content, among them a variety of software reliability growth models (SRGMs). SRGMs have underlying assumptions that are often violated in practice, but empirical evidence has shown that many are quite robust despite these assumption violations. The problem is that, because of assumption violations, it is often difficult to know which models to apply in practice. We present an empirical method for selecting SRGMs to make release decisions. The method provides guidelines on how to select among the SRGMs to decide on the best model to use as failures are reported during the test phase. The method applies various SRGMs iteratively during system test. They are fitted to weekly cumulative failure data and used to estimate the expected remaining number of failures in software after release. If the SRGMs pass proposed criteria, they may then be used to make release decisions. The method is applied in a case study using defect reports from system testing of three releases of a large medical record system to determine how well it predicts the expected total number of failures.

An Empirical Method for Selecting Software Reliability Growth Models

A large number of software reliability growth models (SRGMs) have been proposed during the past 30 years to estimate software reliability measures such as the number of remaining faults, software failure rate, and software reliability. Selection of an optimal SRGM for use in a particular case has been an area of interest for researchers in the field of software reliability. Tools and techniques for software reliability model selection found in the literature cannot be used with high confidence as they use a limited number of model selection criteria. For the first time, we developed a deterministic quantitative model based on a distance based approach (DBA) method, then applied it for evaluation, optimal selection, and ranking of SRGMs. DBA recognizes the need for relative importance of criteria for a given application, without which inter-criterion comparison could not be accomplished. It requires a set of model selection criteria, along with a set of SRGMs, and their level of criteria for optimal selection; and it successfully presents the results in terms of a merit value which is used to rank the SRGMs. We use two distinct, real data sets for demonstration of the DBA method. The result of this study will be a ranking of SRGMs based on the Euclidean composite distance of each alternative to the designated optimal SRGM.

Selection of Optimal Software Reliability Growth Models Using a Distance Based Approach

Flexible software reliability growth model with testing effort dependent learning process

Enhancing software reliability modeling and prediction through the introduction of time-variable fault reduction factor

Selecting software reliability growth models and improving their predictive accuracy using historical projects data

Predicting the remaining errors in a software system historically has been difficult to do with accuracy. The models used to predict future events have often worked well on one system or collection of data, and not at all well on another. Much of the recent work in the software reliability field has been on model selection and identifying which model would work well with which software system. The Akaike Information Criterion can be used to select the best model from among several models. A case study is given of an application of this technique to an ongoing software project. The Akaike Information Criterion was used to select the best model for a system and then that model was used to predict the number of remaining errors. >

T.G. Woodcock

Papers

Software reliability model selection: a cast study