Software fault localization, the act of identifying the locations of faults in a program, is widely recognized to be one of the most tedious, time consuming, and expensive – yet equally critical – activities in program debugging. Due to the increasing scale and complexity of software today, manually locating faults when failures occur is rapidly becoming infeasible, and consequently, there is a strong demand for techniques that can guide software developers to the locations of faults in a program with minimal human intervention. This demand in turn has fueled the proposal and development of a broad spectrum of fault localization techniques, each of which aims to streamline the fault localization process and make it more effective by attacking the problem in a unique way. In this article, we catalog and provide a comprehensive overview of such techniques and discuss key issues and concerns that are pertinent to software fault localization as a whole.

A Survey on Software Fault Localization

We present Prophet, a novel patch generation system that works with a set of successful human patches obtained from open- source software repositories to learn a probabilistic, application-independent model of correct code. It generates a space of candidate patches, uses the model to rank the candidate patches in order of likely correctness, and validates the ranked patches against a suite of test cases to find correct patches. Experimental results show that, on a benchmark set of 69 real-world defects drawn from eight open-source projects, Prophet significantly outperforms the previous state-of-the-art patch generation system.

/pdf/automatic-patch-generation-by-learning-correct-code-2y44ih20iy.pdf

Automatic patch generation by learning correct code

Since debugging is a time-consuming activity, automated program repair tools such as GenProg have garnered interest A recent study revealed that the majority of GenProg repairs avoid bugs simply by deleting functionality We found that SPR, a state-of-the-art repair tool proposed in 2015, still deletes functionality in their many "plausible" repairs Unlike generate-and-validate systems such as GenProg and SPR, semantic analysis based repair techniques synthesize a repair based on semantic information of the program While such semantics-based repair methods show promise in terms of quality of generated repairs, their scalability has been a concern so far In this paper, we present Angelix, a novel semantics-based repair method that scales up to programs of similar size as are handled by search-based repair tools such as GenProg and SPR This shows that Angelix is more scalable than previously proposed semantics based repair methods such as SemFix and DirectFix Furthermore, our repair method can repair multiple buggy locations that are dependent on each other Such repairs are hard to achieve using SPR and GenProg In our experiments, Angelix generated repairs from large-scale real-world software such as wireshark and php, and these generated repairs include multi-location repairs We also report our experience in automatically repairing the well-known Heartbleed vulnerability

/pdf/angelix-scalable-multiline-program-patch-synthesis-via-1loaty358w.pdf

Angelix: scalable multiline program patch synthesis via symbolic analysis

We propose Nopol , an approach to automatic repair of buggy conditional statements (i.e., if-then-else statements). This approach takes a buggy program as well as a test suite as input and generates a patch with a conditional expression as output. The test suite is required to contain passing test cases to model the expected behavior of the program and at least one failing test case that reveals the bug to be repaired. The process of Nopol consists of three major phases. First, Nopol employs angelic fix localization to identify expected values of a condition during the test execution. Second, runtime trace collection is used to collect variables and their actual values, including primitive data types and objected-oriented features (e.g., nullness checks), to serve as building blocks for patch generation. Third, Nopol encodes these collected data into an instance of a Satisfiability Modulo Theory (SMT) problem; then a feasible solution to the SMT instance is translated back into a code patch. We evaluate Nopol on 22 real-world bugs (16 bugs with buggy if conditions and six bugs with missing preconditions) on two large open-source projects, namely Apache Commons Math and Apache Commons Lang. Empirical analysis on these bugs shows that our approach can effectively fix bugs with buggy if conditions and missing preconditions. We illustrate the capabilities and limitations of Nopol using case studies of real bug fixes.

https://hal.archives-ouvertes.fr/hal-01285008/document

Nopol: Automatic Repair of Conditional Statement Bugs in Java Programs

We present SPR, a new program repair system that combines staged program repair and condition synthesis. These techniques enable SPR to work productively with a set of parameterized transformation schemas to generate and efficiently search a rich space of program repairs. Together these techniques enable SPR to generate correct repairs for over five times as many defects as previous systems evaluated on the same benchmark set.

/pdf/staged-program-repair-with-condition-synthesis-3j1nk9738s.pdf

Staged program repair with condition synthesis

Automated program repair recently received considerable attentions, and many techniques on this research area have been proposed. Among them, two genetic-programming-based techniques, GenProg and Par, have shown the promising results. In particular, GenProg has been used as the baseline technique to check the repair effectiveness of new techniques in much literature. Although GenProg and Par have shown their strong ability of fixing real-life bugs in nontrivial programs, to what extent GenProg and Par can benefit from genetic programming, used by them to guide the patch search process, is still unknown. To address the question, we present a new automated repair technique using random search, which is commonly considered much simpler than genetic programming, and implement a prototype tool called RSRepair. Experiment on 7 programs with 24 versions shipping with real-life bugs suggests that RSRepair, in most cases (23/24), outperforms GenProg in terms of both repair effectiveness (requiring fewer patch trials) and efficiency (requiring fewer test case executions), justifying the stronger strength of random search over genetic programming. According to experimental results, we suggest that every proposed technique using optimization algorithm should check its effectiveness by comparing it with random search.

/pdf/the-strength-of-random-search-on-automated-program-repair-29yxr9h46x.pdf

The strength of random search on automated program repair

Slice-based statistical fault localization

Most techniques for automated program repair use test cases to validate the effectiveness of the produced patches. The validation process can be time-consuming especially when the object programs ship with either lots of test cases or some long-running test cases. To alleviate the cost for testing, we first introduce regression test prioritization insight into the area of automated program repair, and present a novel prioritization technique called FRTP with the goal of reducing the number of test case executions in the repair process. Unlike most existing prioritization techniques frequently requiring additional cost for gathering previous test executions information, FRTP iteratively extracts that information just from the repair process, and thus incurs trivial performance lose. We also built a tool called TrpAutoRepair, which implements our FRTP technique and has the ability of automatically repairing C programs. To evaluate TrpAutoRepair, we compared it with GenProg, a state-of-the-art tool for automated C program repair. The experiment on the 5 subject programs with 16 real-life bugs provides evidence that TrpAutoRepair performs at least as good as GenProg in term of success rate, in most cases (15/16), TrpAutoRepair can significantly improve the repair efficiency by reducing efficiently the test case executions when searching a valid patch in the repair process.

Efficient Automated Program Repair through Fault-Recorded Testing Prioritization

Many techniques on automated fault localization (AFL) have been introduced to assist developers in debugging. Prior studies evaluate the localization technique from the viewpoint of developers: measuring how many benefits that developers can obtain from the localization technique used when debugging. However, these evaluation approaches are not always suitable, because it is difficult to quantify precisely the benefits due to the complex debugging behaviors of developers. In addition, recent user studies have presented that developers working with AFL do not correct the defects more efficiently than ones working with only traditional debugging techniques such as breakpoints, even when the effectiveness of AFL is artificially improved. In this paper we attempt to propose a new research direction of developing AFL techniques from the viewpoint of fully automated debugging including the program repair of automation, for which the activity of AFL is necessary. We also introduce the NCP score as the evaluation measurement to assess and compare various techniques from this perspective. Our experiment on 15 popular AFL techniques with 11 subject programs shipping with real-life field failures presents the evidence that these AFL techniques performing well in prior studies do not have better localization effectiveness according to NCP score. We also observe that Jaccard has the better performance over other techniques in our experiment.

Using automated program repair for evaluating the effectiveness of fault localization techniques

Automated program repair has made some important progress in the recent decade. One well-known repair tool is GneProg, which automates the patch generation process according to the guidance of genetic programming. Although GenProg has successfully fixed many legacy faulty programs, the guidance effectiveness of genetic programming used by GenProg has not been even justified through the comparison with random search algorithm. In this paper we try to make the guidance effectiveness comparison between genetic programming and random search algorithm on program repair. The experimental results show that genetic programming does not perform better than random search algorithm on guiding the patch generation process.

Yuhua Qi

Papers

The strength of random search on automated program repair

Slice-based statistical fault localization

Efficient Automated Program Repair through Fault-Recorded Testing Prioritization

Using automated program repair for evaluating the effectiveness of fault localization techniques

Does Genetic Programming Work Well on Automated Program Repair