Hawaii International Conference on System Sciences

The value of using static code attributes to learn defect predictors has been widely debated. Prior work has explored issues like the merits of "McCabes versus Halstead versus lines of code counts" for generating defect predictors. We show here that such debates are irrelevant since how the attributes are used to build predictors is much more important than which particular attributes are used. Also, contrary to prior pessimism, we show that such defect predictors are demonstrably useful and, on the data studied here, yield predictors with a mean probability of detection of 71 percent and mean false alarms rates of 25 percent. These predictors would be useful for prioritizing a resource-bound exploration of code that has yet to be inspected

/pdf/data-mining-static-code-attributes-to-learn-defect-2iiikp0sat.pdf

Data Mining Static Code Attributes to Learn Defect Predictors

Background: The accurate prediction of where faults are likely to occur in code can help direct test effort, reduce costs, and improve the quality of software. Objective: We investigate how the context of models, the independent variables used, and the modeling techniques applied influence the performance of fault prediction models. Method: We used a systematic literature review to identify 208 fault prediction studies published from January 2000 to December 2010. We synthesize the quantitative and qualitative results of 36 studies which report sufficient contextual and methodological information according to the criteria we develop and apply. Results: The models that perform well tend to be based on simple modeling techniques such as Naive Bayes or Logistic Regression. Combinations of independent variables have been used by models that perform well. Feature selection has been applied to these combinations when models are performing particularly well. Conclusion: The methodology used to build models seems to be influential to predictive performance. Although there are a set of fault prediction studies in which confidence is possible, more studies are needed that use a reliable methodology and which report their context, methodology, and performance comprehensively.

/pdf/a-systematic-literature-review-on-fault-prediction-50xbihfieh.pdf

A Systematic Literature Review on Fault Prediction Performance in Software Engineering

To reduce the cost of regression testing, software testers may prioritize their test cases so that those which are more important, by some measure, are run earlier in the regression testing process. One potential goal of such prioritization is to increase a test suite's rate of fault detection. Previous work reported results of studies that showed that prioritization techniques can significantly improve rate of fault detection. Those studies, however, raised several additional questions: 1) Can prioritization techniques be effective when targeted at specific modified versions; 2) what trade-offs exist between fine granularity and coarse granularity prioritization techniques; 3) can the incorporation of measures of fault proneness into prioritization techniques improve their effectiveness? To address these questions, we have performed several new studies in which we empirically compared prioritization techniques using both controlled experiments and case studies.

/pdf/test-case-prioritization-a-family-of-empirical-studies-5d19ow5fgj.pdf

Test case prioritization: a family of empirical studies

Test case prioritization techniques schedule test cases in an order that increases their effectiveness in meeting some performance goal. One performance goal, rate of fault detection, is a measure of how quickly faults are detected within the testing process; an improved rate of fault detection can provide faster feedback on the system under test, and let software engineers begin locating and correcting faults earlier than might otherwise be possible. In previous work, we reported the results of studies that showed that prioritization techniques can significantly improve rate of fault detection. Those studies, however, raised several additional questions: (1) can prioritization techniques be effective when aimed at specific modified versions; (2) what tradeoffs exist between fine granularity and coarse granularity prioritization techniques; (3) can the incorporation of measures of fault proneness into prioritization techniques improve their effectiveness? This paper reports the results of new experiments addressing these questions.

/pdf/prioritizing-test-cases-for-regression-testing-31qulwuphm.pdf

Prioritizing test cases for regression testing

As space mission software becomes more complex, the ability to effectively deal with faults is increasingly important The strategies that can be employed for fighting a software bug depend on its fault type Bohrbugs are easily isolated and removed during software testing Mandelbugs appear to behave chaotically While it is more difficult to detect these faults during testing, it may not be necessary to correct them; a simple retry after a failure occurrence may work Aging-related bugs, a sub-class of Mandelbugs, can cause an increasing failure rate For these faults, proactive techniques may prevent future failures In this paper, we analyze the faults discovered in the on-board software for 18 JPL/NASA space missions We present the proportions of the various fault types and study how they have evolved over time Moreover, we examine whether or not the fault type and attributes such as the failure effect are independent

https://grottke.de/documents/anempiricalinvestigationoffaulttypes.pdf

An empirical investigation of fault types in space mission system software

A set of linear combination software reliability models that combine the results of single, or component, models is presented. It is shown that, as measured by statistical methods for determining a model's applicability to a set of failure data, a combination model tends to have more accurate short-term and long-term predictions than a component model. These models were evaluated using both historical data sets and data from recent Jet Propulsion Laboratory projects. The computer-aided software reliability estimation (CASRE) tool, which automates many reliability measurement tasks and makes it easier to apply reliability models and to form combination models, is described. >

Applying reliability models more effectively (software)

Over the past several years, we have been developing methods of predicting the fault content of software systems based on measured characteristics of their structural evolution. In previous work, we have shown there is a significant linear relationship between code churn, a synthesized metric, and the rate at which faults are inserted into the system in terms of number of faults per unit change in code churn. We have begun a new investigation of this relationship with a flight software technology development effort at the jet propulsion laboratory (JPL) and have progressed in resolving the limitations of the earlier work in two distinct steps. First, we have developed a standard for the enumeration of faults. Second, we have developed a practical framework for automating the measurement of these faults. we analyze the measurements of structural evolution and fault counts obtained from the JPL flight software technology development effort. Our results indicate that the measures of structural attributes of the evolving software system are suitable for forming predictors of the number of faults inserted into software modules during their development. The new fault standard also ensures that the model so developed has greater predictive validity.

Developing fault predictors for evolving software systems

The construction of a computer-aided software engineering (CASE) tool for a systematic and automatic application of software reliability modeling for real-world projects is described. Emphasis is placed on the practical project applications of existing software reliability models for better software reliability estimations. This CASE tool is built on top of existing software reliability models, called component models, as the baseline for software reliability measurement. The main advantage of this tool over other similar tools is its various reliability estimations under a paradigm which linearly combines the component models. This tool features an enhanced graphical user-interface which greatly facilitates the potentially tedious application procedure for software reliability estimation. >

CASRE: a computer-aided software reliability estimation tool

Analyzing requirements for consistency and checking them for correctness can require significant effort, particularly if they have not been maintained with a requirements management tool (e.g., DOORS) or specified in a machine-readable notation. By restricting the number of requirements being analyzed, fewer opportunities exist for introducing errors into the analysis. This can be accomplished by subsetting the requirements and analyzing one subset at a time.Previous work showed that simple natural language processing and machine learning techniques can be used to identify temporal requirements within a set of natural language requirements. This paper builds on that work by detailing our results in applying these techniques to a set of natural-language temporal requirements taken from a current JPL mission and determining whether a requirement is one of the most frequently occurring types of temporal requirements.The ability to distinguish between different LTL patterns in natural-language requirements raises the possibility of automating the transformation of natural-language temporal requirements into LTL expressions. This would allow automated consistency checking and tracing of natural-language temporal requirements. Since correctness properties are often specified as LTL expressions, this would also provide a set of correctness properties against which abstract models of the system could be verified.

Allen P. Nikora

Papers

An empirical investigation of fault types in space mission system software

Applying reliability models more effectively (software)

Developing fault predictors for evolving software systems

CASRE: a computer-aided software reliability estimation tool

Automated Identification of LTL Patterns in Natural Language Requirements