Software fault localization, the act of identifying the locations of faults in a program, is widely recognized to be one of the most tedious, time consuming, and expensive – yet equally critical – activities in program debugging. Due to the increasing scale and complexity of software today, manually locating faults when failures occur is rapidly becoming infeasible, and consequently, there is a strong demand for techniques that can guide software developers to the locations of faults in a program with minimal human intervention. This demand in turn has fueled the proposal and development of a broad spectrum of fault localization techniques, each of which aims to streamline the fault localization process and make it more effective by attacking the problem in a unique way. In this article, we catalog and provide a comprehensive overview of such techniques and discuss key issues and concerns that are pertinent to software fault localization as a whole.

A Survey on Software Fault Localization

This article presents an improved approach to assist diagnosis of failures in software (fault localisation) by ranking program statements or blocks in accordance with to how likely they are to be buggy. We present a very simple single-bug program to model the problem. By examining different possible execution paths through this model program over a number of test cases, the effectiveness of different proposed spectral ranking methods can be evaluated in idealised conditions. The results are remarkably consistent to those arrived at empirically using the Siemens test suite and Space benchmarks. The model also helps identify groups of metrics that are equivalent for ranking. Due to the simplicity of the model, an optimal ranking method can be devised. This new method out-performs previously proposed methods for the model program, the Siemens test suite and Space. It also helps provide insight into other ranking methods.

/pdf/a-model-for-spectra-based-software-diagnosis-4eia6eyyxq.pdf

A model for spectra-based software diagnosis

We propose Nopol , an approach to automatic repair of buggy conditional statements (i.e., if-then-else statements). This approach takes a buggy program as well as a test suite as input and generates a patch with a conditional expression as output. The test suite is required to contain passing test cases to model the expected behavior of the program and at least one failing test case that reveals the bug to be repaired. The process of Nopol consists of three major phases. First, Nopol employs angelic fix localization to identify expected values of a condition during the test execution. Second, runtime trace collection is used to collect variables and their actual values, including primitive data types and objected-oriented features (e.g., nullness checks), to serve as building blocks for patch generation. Third, Nopol encodes these collected data into an instance of a Satisfiability Modulo Theory (SMT) problem; then a feasible solution to the SMT instance is translated back into a code patch. We evaluate Nopol on 22 real-world bugs (16 bugs with buggy if conditions and six bugs with missing preconditions) on two large open-source projects, namely Apache Commons Math and Apache Commons Lang. Empirical analysis on these bugs shows that our approach can effectively fix bugs with buggy if conditions and missing preconditions. We illustrate the capabilities and limitations of Nopol using case studies of real bug fixes.

https://hal.archives-ouvertes.fr/hal-01285008/document

Nopol: Automatic Repair of Conditional Statement Bugs in Java Programs

Most fault localization techniques take as input a faulty program, and produce as output a ranked list of suspicious code locations at which the program may be defective When researchers propose a new fault localization technique, they typically evaluate it on programs with known faults The technique is scored based on where in its output list the defective code appears This enables the comparison of multiple fault localization techniques to determine which one is better Previous research has evaluated fault localization techniques using artificial faults, generated either by mutation tools or manually In other words, previous research has determined which fault localization techniques are best at finding artificial faults However, it is not known which fault localization techniques are best at finding real faults It is not obvious that the answer is the same, given previous work showing that artificial faults have both similarities to and differences from real faults We performed a replication study to evaluate 10 claims in the literature that compared fault localization techniques (from the spectrum-based and mutation-based families) We used 2995 artificial faults in 6 real-world programs Our results support 7 of the previous claims as statistically significant, but only 3 as having non-negligible effect sizes Then, we evaluated the same 10 claims, using 310 real faults from the 6 programs Every previous result was refuted or was statistically and practically insignificant Our experiments show that artificial faults are not useful for predicting which fault localization techniques perform best on real faults In light of these results, we identified a design space that includes many previously-studied fault localization techniques as well as hundreds of new techniques We experimentally determined which factors in the design space are most important, using an overall set of 395 real faults Then, we extended this design space with new techniques Several of our novel techniques outperform all existing techniques, notably in terms of ranking defective code in the top-5 or top-10 reports

Evaluating and improving fault localization

Effective debugging is crucial to producing reliable software. Manual debugging is becoming prohibitively expensive, especially due to the growing size and complexity of programs. Given that fault localization is one of the most expensive activities in program debugging, there has been a great demand for fault localization techniques that can help guide programmers to the locations of faults. In this paper, a technique named DStar (D*) is proposed which can suggest suspicious locations for fault localization automatically without requiring any prior information on program structure or semantics. D* is evaluated across 24 programs, and is compared to 38 different fault localization techniques. Both single-fault and multi-fault programs are used. Results indicate that D* is more effective at locating faults than all the other techniques it is compared to. An empirical evaluation is also conducted to illustrate how the effectiveness of D* increases as the exponent * grows, and then levels off when the exponent * exceeds a critical value. Discussions are presented to support such observations.

The DStar Method for Effective Software Fault Localization

In practice, manual debugging to locate bugs is a daunting and time-consuming task. By using software fault localization, we can reduce this time substantially. The technique of software fault localization can be performed using execution profiles of the software under several test inputs. Such profiles, known as program spectra, consist of the coverage of correct and incorrect executions statement from a given test suite. We have performed a systematic evaluation of several metrics that make use of measurement obtained from program spectra on Siemens Test Suite. In this paper, we discuss how the effectiveness of various metrics degrade in determining buggy statements as the bug consistency (error detection accuracy, q e ) of a statement approaches zero. Bug consistency of a statement refers to the ratio of the number of failed tests executing the statement over the total number of tests executing the statement. We proposed Effect(M) as to measure the effectiveness of these metrics as q e value varies. We also demonstrate that the q e (previously not considered as a metric), is just as effective as some of the metrics proposed. We also formally prove that q e is identical to the metric that Tarantula system uses for bug localization.

Study of the relationship of bug consistency with respect to performance of spectra metrics

Software faults can be diagnosed using program spectra. The program spectra considered here provide information about which statements are executed in each one of a set of test cases. This information is used to compute a value for each statement which indicates how likely it is to be buggy, and the statements are ranked according to these values. We present two improvements to this method. First, we associate varying weights with failed test cases --- test cases which execute fewer statements are given more weight and have more influence on the ranking. This generally improves diagnosis accuracy, with little additional cost. Second, the ranking is computed incrementally. After the top-ranked statement is identified, the weights are adjusted in order to compute the rest of the ranking. This further improves accuracy. The cost is more significant, but not prohibitive.

/pdf/spectral-debugging-with-weights-and-incremental-ranking-27pxrrozpc.pdf

Spectral Debugging with Weights and Incremental Ranking

Numerous set similarity metrics have been used for ranking "suspiciousness" of code in spectral fault localization, which uses execution profiles of passed and failed test cases to help locate bugs. Research in data mining has identified several forms of possibly desirable symmetry in similarity metrics. Here we define several forms of "duals" of metrics, based on these forms of symmetries. Use of these duals, plus some other slight modifications, leads to several new similarity metrics. We show that versions of several previously proposed metrics are optimal, or nearly optimal, for locating single bugs. We also show that a form of duality exists between locating single bugs and locating "deterministic" bugs (execution of which always results in test case failure). Duals of the various single bug optimal metrics are optimal for locating such bugs. This more theoretical work leads to a conjecture about how different metrics could be chosen for different stages of software development.

/pdf/duals-in-spectral-fault-localization-3pzhvy1qp5.pdf

Duals in Spectral Fault Localization

This paper presents an approach of bug localization using a frequency weighting function. In an existing approach, only binary information of execution count from test executions is used. Information of each program statement being executed and not executed by a particular test is used; indicated by 1 and 0 respectively. In our proposed approach, frequency execution count of each program statement executed by a respective test is used. We evaluate several well-known spectra metrics using our proposed approach and the existing approach (using binary information of execution count) on two test suites; Siemens Test Suite and Unix datasets. We show that the bug localization performance is improved by using our proposed approach. We conduct statistical test and show that the improved bug localization performance using our approach (using frequency execution count) is statistically significant than using the existing approach (using binary information of execution count).

Hua Jie Lee

Papers

A model for spectra-based software diagnosis

Study of the relationship of bug consistency with respect to performance of spectra metrics

Spectral Debugging with Weights and Incremental Ranking

Duals in Spectral Fault Localization

Effective Software Bug Localization Using Spectral Frequency Weighting Function