In the past five years there has been a dramatic increase in work on Search-Based Software Engineering (SBSE), an approach to Software Engineering (SE) in which Search-Based Optimization (SBO) algorithms are used to address problems in SE. SBSE has been applied to problems throughout the SE lifecycle, from requirements and project planning to maintenance and reengineering. The approach is attractive because it offers a suite of adaptive automated and semiautomated solutions in situations typified by large complex problem spaces with multiple competing and conflicting objectives.This article1 provides a review and classification of literature on SBSE. The work identifies research trends and relationships between the techniques applied and the applications to which they have been applied and highlights gaps in the literature and avenues for further research.

/pdf/search-based-software-engineering-trends-techniques-and-3b0bwaagfi.pdf

Search-based software engineering: Trends, techniques and applications

A good test suite is one that detects real faults Because the set of faults in a program is usually unknowable, this definition is not useful to practitioners who are creating test suites, nor to researchers who are creating and evaluating tools that generate test suites In place of real faults, testing research often uses mutants, which are artificial faults -- each one a simple syntactic variation -- that are systematically seeded throughout the program under test Mutation analysis is appealing because large numbers of mutants can be automatically-generated and used to compensate for low quantities or the absence of known real faults Unfortunately, there is little experimental evidence to support the use of mutants as a replacement for real faults This paper investigates whether mutants are indeed a valid substitute for real faults, ie, whether a test suite’s ability to detect mutants is correlated with its ability to detect real faults that developers have fixed Unlike prior studies, these investigations also explicitly consider the conflating effects of code coverage on the mutant detection rate Our experiments used 357 real faults in 5 open-source applications that comprise a total of 321,000 lines of code Furthermore, our experiments used both developer-written and automatically-generated test suites The results show a statistically significant correlation between mutant detection and real fault detection, independently of code coverage The results also give concrete suggestions on how to improve mutation analysis and reveal some inherent limitations

/pdf/are-mutants-a-valid-substitute-for-real-faults-in-software-2w3vlgaztq.pdf

Are mutants a valid substitute for real faults in software testing

This paper defines software fairness and discrimination and develops a testing-based method for measuring if and how much software discriminates, focusing on causality in discriminatory behavior. Evidence of software discrimination has been found in modern software systems that recommend criminal sentences, grant access to financial products, and determine who is allowed to participate in promotions. Our approach, Themis, generates efficient test suites to measure discrimination. Given a schema describing valid system inputs, Themis generates discrimination tests automatically and does not require an oracle. We evaluate Themis on 20 software systems, 12 of which come from prior work with explicit focus on avoiding discrimination. We find that (1) Themis is effective at discovering software discrimination, (2) state-of-the-art techniques for removing discrimination from algorithms fail in many situations, at times discriminating against as much as 98% of an input subdomain, (3) Themis optimizations are effective at producing efficient test suites for measuring discrimination, and (4) Themis is more efficient on systems that exhibit more discrimination. We thus demonstrate that fairness testing is a critical aspect of the software development cycle in domains with possible discrimination and provide initial tools for measuring software discrimination.

https://dl.acm.org/doi/pdf/10.1145/3106237.3106277

Fairness testing: testing software for discrimination

In the past five years there has been a dramatic increase in work on Search Based Software Engineering (SBSE), an approach to software engineering in which search based optimisation algorithms are used to address problems in Software Engineering. SBSE has been applied to problems throughout the Software Engineering lifecycle, from requirements and project planning to maintenance and re-engineering. The approach is attractive because it offers a suite of adaptive automated and semi-automated solutions in situations typified by large complex problem spaces with multiple competing and conflicting objectives. This paper provides a review and classification of literature on SBSE. The paper identifies research trends and relationships between the techniques applied and the applications to which they have been applied and highlights gaps in the literature and avenues for further research.

/pdf/search-based-software-engineering-a-comprehensive-analysis-h98ne3mmop.pdf

Search Based Software Engineering: A Comprehensive Analysis and Review of Trends Techniques and Applications

To assess the quality of test suites, mutation analysis seeds artificial defects (mutations) into programs; a nondetected mutation indicates a weakness in the test suite We present an automated approach to generate unit tests that detect these mutations for object-oriented classes This has two advantages: First, the resulting test suite is optimized toward finding defects modeled by mutation operators rather than covering code Second, the state change caused by mutations induces oracles that precisely detect the mutants Evaluated on 10 open source libraries, our μtest prototype generates test suites that find significantly more seeded defects than the original manually written test suites

/pdf/mutation-driven-generation-of-unit-tests-and-oracles-1cjln5j969.pdf

Mutation-Driven Generation of Unit Tests and Oracles

Software programs evolve throughout their lifetime undergoing various changes. While making these changes, software developers may introduce regression faults. It is desirable to detect these faults as quickly as possible to reduce the cost involved in fixing them. One existing solution is continuous testing, which runs an existing test suite to quickly find regression faults as soon as code changes are saved. However, the effectiveness of continuous testing depends on the capability of the existing test suite for finding behavioral differences across versions.To address the issue, we propose an approach, called eXpress, that conducts efficient regression test generation based on a path-exploration-based test generation (PBTG) technique, such as dynamic symbolic execution. eXpress prunes various irrelevant paths with respect to detecting behavioral differences to optimize the search strategy of a PBTG technique. As a result, the PBTG technique focuses its efforts on regression test generation. In addition, eXpress leverages the existing test suite (if available) for the original version to efficiently execute the changed code regions of the program and infect program states. Experimental results on 67 versions (in total) of four programs (two from the subject infrastructure repository and two from real-world open source projects) show that, using eXpress, a state-of-the-art PBTG technique, called Pex, requires about 36% less amount of time (on average) to detect behavioral differences than without using eXpress. In addition, Pex using eXpress detects four behavioral differences that could not be detected without using eXpress (within a time bound). Furthermore, Pex requires 67% less amount of time to find behavioral differences by exploiting an existing test suite than exploration without using the test suite.

/pdf/express-guided-path-exploration-for-efficient-regression-u1g74s1o4u.pdf

eXpress: guided path exploration for efficient regression test generation

Software testing has been commonly used in assuring the quality of database applications. It is often prohibitively expensive to manually write quality tests for complex database applications. Automated test generation techniques, such as Dynamic Symbolic Execution (DSE), have been proposed to reduce human efforts in testing database applications. However, such techniques have two major limitations: (1) they assume that the database that the application under test interacts with is accessible, which may not always be true; and (2) they usually cannot create necessary database states as a part of the generated tests.To address the preceding limitations, we propose an approach that applies DSE to generate tests for a database application. Instead of using the actual database that the application interacts with, our approach produces and uses a mock database in test generation. A mock database mimics the behavior of an actual database by performing identical database operations on itself. We conducted two empirical evaluations on both a medical device and an open source software system to demonstrate that our approach can generate, without producing false warnings, tests with higher code coverage than conventional DSE-based techniques.

/pdf/moda-automated-test-generation-for-database-applications-via-3xozl89xcp.pdf

MODA: automated test generation for database applications via mock objects

Software programs continue to evolve throughout their lifetime. Maintenance of such evolving programs, including regression testing, is one of the most expensive activities in software development. We present an approach and its implementation called DiffGen for automated regression unit-test generation and checking for Java programs. Given two versions of a Java class, our approach instruments the code by adding new branches such that if these branches can be covered by a test generation tool, behavioral differences between the two class versions are exposed. DiffGen then uses a coverage-based test generation tool to generate test inputs for covering the added branches to expose behavioral differences. We have evaluated DiffGen on finding behavioral differences between 21 classes and their versions. Experimental results show that our approach can effectively expose many behavioral differences that cannot be exposed by state-of-the-art techniques.

/pdf/diffgen-automated-regression-unit-test-generation-26815ol7p8.pdf

DiffGen: Automated Regression Unit-Test Generation

Software-defect detection is an increasingly important research topic in software engineering. To detect defects in a program, static verification and dynamic test generation are two important proposed techniques. However, both of these techniques face their respective issues. Static verification produces false positives, and on the other hand, dynamic test generation is often time consuming. To address the limitations of static verification and dynamic test generation, we present an automated defect-detection tool, called DyTa, that combines both static verification and dynamic test generation. DyTa consists of a static phase and a dynamic phase. The static phase detects potential defects with a static checker; the dynamic phase generates test inputs through dynamic symbolic execution to confirm these potential defects. DyTa reduces the number of false positives compared to static verification and performs more efficiently compared to dynamic test generation.

/pdf/dyta-dynamic-symbolic-execution-guided-with-static-4rtlmy4cu2.pdf

DyTa: dynamic symbolic execution guided with static verification results

No consolidated set of software engineering best practices for the Internet of Things (IoT) has yet emerged. Too often, the landscape resembles the Wild West, with unprepared programmers putting together IoT systems in ad hoc fashion and throwing them out into the market, often poorly tested. In addition, the academic sector is in danger of fragmenting into specialized, often unrelated research areas. This IEEE Software theme issue aims to help provide the basis for a set of best practices that will guide the industry through the challenges of software engineering for the IoT

Kunal Taneja

Papers

eXpress: guided path exploration for efficient regression test generation

MODA: automated test generation for database applications via mock objects

DiffGen: Automated Regression Unit-Test Generation

DyTa: dynamic symbolic execution guided with static verification results

Software Engineering for the Internet of Things