PSO based test case generation for critical path using improved combined fitness function

This is the peer reviewed version which has been published in final form at [DOI]. This article may be used for non-commercial purposes in accordance with Wiley Terms and Conditions for Use of Self-Archived Versions.

/pdf/branch-coverage-prediction-in-automated-testing-4k55pf1ytc.pdf

Branch coverage prediction in automated testing

Context Search-based Software Testing (SBST) is a research field where testing a software product is formulated as an optimization problem. It is an active sub-area of Search-based Software Engineering (SBSE) where many studies have been published and some reviews have been carried out. The majority of studies in SBST has been adopted meta-heuristics while hyper-heuristics have a long way to go. Moreover, there is still a lack of studies to perceive the state-of-the-art of the use of hyper-heuristics within SBST. Objective The objective of this work is to investigate the adoption of hyper-heuristics for Software Testing highlighting the current efforts and identifying new research directions. Method A Systematic mapping study was carried out with 5 research questions considering papers published up to may/2019, and 4 different bases. The research questions aims to find out, among other things, what are the hyper-heuristics used in the context of Software Testing, for what problems hyper-heuristics have been applied, and what are the objective functions in the scope of Software Testing. Results A total of 734 studies were found via the search strings and 164 articles were related to Software Testing. However, from these, only 26 papers were actually in accordance with the scope of this research and 3 more papers were considered due to snowballing or expert’s suggestion, totalizing 29 selected papers. Few different problems and application domains where hyper-heuristics have been considered were identified. Conclusion Differently from other communities (Operational Research, Artificial Intelligence), SBST has little explored the benefits of hyper-heuristics which include generalization and less difficulty in parameterization. Hence, it is important to further investigate this area in order to alleviate the effort of practitioners to use such an approach in their testing activities.

A systematic mapping addressing Hyper-Heuristics within Search-based Software Testing

In Search-based Software Testing (SBST), test generation is guided by fitness functions that estimate how close a test case is to reach an uncovered test goal (e.g., branch). A popular fitness function estimates how close conditional statements are to evaluating to true or false, i.e., the branch distance. However, when conditions read Boolean variables (e.g., if(x && y)), the branch distance provides no gradient for the search, since a Boolean can either be true or false. This flag problem can be addressed by transforming individual procedures such that Boolean flags are replaced with numeric comparisons that provide better guidance for the search. Unfortunately, defining a semantics-preserving transformation that is applicable in an interprocedural case, where Boolean flags are passed around as parameters and return values, is a daunting task. Thus, it is not yet supported by modern test generators. This work is based on the insight that fitness gradients can be recovered by using runtime information: Given an uncovered interprocedural flag branch, our approach (1) calculates context-sensitive branch distance for all control flows potentially returning the required flag in the called method, and (2) recursively aggregates these distances into a continuous value. We implemented our approach on top of the EvoSuite framework for Java, and empirically compared it with state-of-the-art testability transformations on non-trivial methods suffering from interprocedural flag problems, sampled from open source Java projects. Our experiment demonstrates that our approach achieves higher coverage on the subject methods with statistical significance and acceptable runtime overheads.

https://dl.acm.org/doi/pdf/10.1145/3395363.3397358

Recovering fitness gradients for interprocedural Boolean flags in search-based testing

The multiobjective testing resource allocation problem (MOTRAP) is how to efficiently allocate the finite testing time to various modules, with the aim of optimizing system reliability, testing cost, and testing time simultaneously To deal with this problem, a common approach is to use multiobjective evolutionary algorithms (MOEAs) to seek a set of tradeoff solutions between the three objectives However, such a tradeoff set may contain a substantial proportion of solutions with very low reliability level, which consume lots of computational resources but may be valueless to the software project manager In this article, a MOTRAP model with a prespecified reliability is first proposed Then, new lower bounds on the testing time invested in different modules are theoretically deduced from the necessary condition for the achievement of the given reliability, based on which an exact algorithm for determining the new lower bounds is presented Moreover, several enhanced constraint-handling techniques (ECHTs) derived from the new bounds are successively developed to be combined with MOEAs to correct and reduce the constraint violation Finally, the proposed ECHTs are evaluated in comparison with various state-of-the-art constraint-solving approaches The comparative results demonstrate that the proposed ECHTs can work well with MOEAs, make the search focus on the feasible region of the prespecified reliability, and provide the software project manager with better and more diverse, satisfactory choices in test planning

Enhanced Constraint Handling for Reliability-Constrained Multiobjective Testing Resource Allocation

Search based software testing has received great attention as a means of automating the test data generation, and the goal is to improve various criteria. There are different types of coverage criteria. In this paper, we deal with the path coverage. Concretely, we focus on the path that is the most difficult to cover. One major limitation of search based testing is the inefficient and insufficiently informed fitness function. To address this problem, we propose an adaptive fitness function based on branch hardness. The branch hardness is measured by the expected number of visits of each branch in the program, which is modeled by an absorbing discrete time Markov chain. By tuning the parameters of branch hardness heuristically, the search hardness, evaluated by the variation coefficient of the fitness function, of generating test data can be minimized. Therefore, this new fitness function is more flexible than the traditional counterparts. In addition, we point out that the present definition of branch distance and the use of normalizing functions are problematic, and propose some improvements. Finally, the empirical study reveals the promising result of our proposal in this paper.

An adaptive fitness function based on branch hardness for search based testing

Search-based software testing has achieved great attention recently, but the efficiency is still the bottleneck of it. This paper focuses on improving the efficiency of generating test data for multiple paths. Genetic algorithms are chosen as the heuristic algorithms in search-based software testing in this paper. First, we propose an improved grouping strategy of target paths to balance the load of each calculation resource. This work makes a contribution to the parallel execution in search-based software testing. Then, common constraints of the target paths in the same group are collected to reduce the search space of test data. Symbolic execution technique is used in this phase. Based on the reduced search space, we can accelerate the convergence of search process and improve the efficiency of search-based software testing. Finally, our method is applied to some study cases to compare with other methods.

Improved evolutionary generation of test data for multiple paths in search-based software testing

Search-based software testing (SBST) has achieved great attention as an effective technique to automate test data generation. The testing problem is converted into a search problem, and a meta-heuristic algorithm is used to search for the test data in SBST. Genetic Algorithm (GA) is the most popular meta-heuristic algorithm used in SBST and the genetic operators are the key parts in GA. Much work has been done to improve SBST while little research has concentrated on the genetic operators. Due to the blindness and randomness of classic genetic operators, SBST is ineffective in many cases. In this paper, we focus on improving the genetic operators by constraint-based software testing. Compared with classic genetic operators, our improved genetic operators are more purposeful. For the selection operator, we use symbolic execution technique to help us select the test cases which have more useful heuristic information. Then, the constraint-based crossover operator recombines the test cases which have more probability to create better offspring individuals. Finally, the constraint-based mutation operator is used to improve the test cases in order to satisfy some specific constraints. We applied our constraint-based genetic operators in several benchmarks and the experiments reveal the promising results of our proposal.

Improving search-based software testing by constraint-based genetic operators

In the area of software testing, search-based software testing (SBST) and dynamic symbolic execution (DSE) are two efficient testing techniques for test cases generation. However, both of the two approaches have their own drawbacks: The efficiency of SBST depends on the guidance of the fitness landscape. When the fitness landscape has some plateaus with no gradient for directing search process, SBST may degenerate into random testing. DSE relies on the capability of constraint solvers. It may struggle to generate test cases with constraints that are difficult to be solved. In this paper, we combine the strengths of both techniques. SBST is used to help DSE for solving difficult constraints and DSE is used to improve the efficiency and capability of SBST. Evolvability metric is introduced for measuring when the software is not suitable for SBST. A novel switch mechanism based on the evolvability metric between SBST and DSE is proposed in this paper to help to choose the proper technique at the proper time. Experiments on several benchmarks reveal the promising results of our proposal.

Combining Search-Based Testing and Dynamic Symbolic Execution by Evolvability Metric

Search Based Software Testing (SBST) formulates testing as an optimization problem, hence some search algorithms can be used to tackle it. The goal of SBST is to improve various test adequacy criteria. There are different types of coverage criteria, and in this paper, we deal with branch coverage, as it is the mostly used criterion. However, the state of the art fitness function definitions used for branch coverage testing have changed little. To fill this gap, this paper proposes a novel fitness function for branch coverage. Concretely, we first use a negative exponential model to evaluate the hardness of covering essential branches of the program, based on which we approximately evaluate the distance of a test candidate to the target branch. Finally, the experiment reveals the promising results of our proposal.

Li Jiao

Papers

An adaptive fitness function based on branch hardness for search based testing

Improved evolutionary generation of test data for multiple paths in search-based software testing

Improving search-based software testing by constraint-based genetic operators

Combining Search-Based Testing and Dynamic Symbolic Execution by Evolvability Metric

A dynamic fitness function for search based software testing