The use of metaheuristic search techniques for the automatic generation of test data has been a burgeoning interest for many researchers in recent years. Previous attempts to automate the test generation process have been limited, having been constrained by the size and complexity of software, and the basic fact that in general, test data generation is an undecidable problem. Metaheuristic search techniques oer much promise in regard to these problems. Metaheuristic search techniques are highlevel frameworks, which utilise heuristics to seek solutions for combinatorial problems at a reasonable computational cost. To date, metaheuristic search techniques have been applied to automate test data generation for structural and functional testing; the testing of grey-box properties, for example safety constraints; and also non-functional properties, such as worst-case execution time. This paper surveys some of the work undertaken in this eld, discussing possible new future directions of research for each of its dieren t individual areas.

https://www.inf.utfsm.cl/~mcriff/Tesistas/Testing/sbs-survey.pdf

Search‐based software test data generation: a survey

Extensively class tested, this text takes an innovative approach to explaining the process of software testing: it defines testing as the process of applying a few well-defined, general-purpose test criteria to a structure or model of the software. The structure of the text directly reflects the pedagogical approach and incorporates the latest innovations in testing, including techniques to test modern types of software such as OO, web applications, and embedded software.

Introduction to Software Testing: List of Figures

Software engineering comprehends several disciplines devoted to prevent and remedy malfunctions and to warrant adequate behaviour. Testing, the subject of this paper, is a widespread validation approach in industry, but it is still largely ad hoc, expensive, and unpredictably effective. Indeed, software testing is a broad term encompassing a variety of activities along the development cycle and beyond, aimed at different goals. Hence, software testing research faces a collection of challenges. A consistent roadmap of the most relevant challenges to be addressed is here proposed. In it, the starting point is constituted by some important past achievements, while the destination consists of four identified goals to which research ultimately tends, but which remain as unreachable as dreams. The routes from the achievements to the dreams are paved by the outstanding research challenges, which are discussed in the paper along with interesting ongoing work.

/pdf/software-testing-research-achievements-challenges-dreams-et7gocee7m.pdf

Software Testing Research: Achievements, Challenges, Dreams

In the past five years there has been a dramatic increase in work on Search-Based Software Engineering (SBSE), an approach to Software Engineering (SE) in which Search-Based Optimization (SBO) algorithms are used to address problems in SE. SBSE has been applied to problems throughout the SE lifecycle, from requirements and project planning to maintenance and reengineering. The approach is attractive because it offers a suite of adaptive automated and semiautomated solutions in situations typified by large complex problem spaces with multiple competing and conflicting objectives.This article1 provides a review and classification of literature on SBSE. The work identifies research trends and relationships between the techniques applied and the applications to which they have been applied and highlights gaps in the literature and avenues for further research.

/pdf/search-based-software-engineering-trends-techniques-and-3b0bwaagfi.pdf

Search-based software engineering: Trends, techniques and applications

An orchestrated survey of methodologies for automated software test case generation

The most important prerequisite for a thorough software test is the determination of relevant test cases. The classification-tree method suggested in this paper supports the systematic determination and description of test cases. It is based on the idea of partition testing. By means of the classification-tree method, the input domain of a test object is regarded under various aspects assessed as relevant by the tester. For each aspect, disjoint and complete classifications are formed. Classes resulting from these classifications may be further classified—even recursively. Test cases are formed by combining classes of different classifications. The stepwise partition of the input domain by means of classifications is represented graphically in the form of a tree. This tree is subsequently used to form a combination table in which the test cases are marked. Extensions of the graphical notation and tool support allow the use of the method even for extensive test problems.

Classification trees for partition testing

Many industrial products are based on the use of embedded computer systems. Usually, these systems have to fulfil real-time requirements, and correct system functionality depends on their logical correctness as well as on their temporal correctness. In order to verify the temporal behavior of real-time systems, previous scientific work has, to a large extent, concentrated on static analysis techniques. Although these techniques offer the possibilty of providing safe estimates of temporal behavior for certain cases, there are a number of cases in practice for which static analysis can not be easily applied. Furthermore, no commercial tools for timing analysis of real-world programs are available. Therefore, the developed systems have to be thoroughly tested in order to detect existing deficiencies in temporal behavior, as well as to strengthen the confidence in temporal correctness. An investigation of existing test methods shows that they mostly concentrate on testing for logical correctness. They are not specialised in the examination of temporal correctness which is also essential to real-time systems. For this reason, existing test procedures must be supplemented by new methods which concentrate on determining whether the system violates its specified timing constraints. Normally, a violation means that outputs are produced too early, or their computation takes too long. The task of the tester therefore is to find the input situations with the longest or shortest execution times, in order to check whether they produce a temporal error. If the search for such inputs is interpreted as a problem of optimization, evolutionary computation can be used to automatically find the inputs with the longest or shortest execution times. This automatic search for accurate test data by means of evolutionary computation is called evolutionary testing. Experiments using evolutionary testing on a number of programs with up to 1511 LOC and 5000 input parameters have successfully identified new longer and shorter execution times than had been found using other testing techniques. Evolutionary testing, therefore, seems to be a promising approach for the verification of timing constraints. A combination of evolutionary testing and systematic testing offers further opportunities to improve the test quality, and could lead to an effective test strategy for real-time systems.

Verifying Timing Constraints of Real-Time Systems by Means of Evolutionary Testing

The systematic test is an inevitable part of the verification and validation process for software. The most important prerequisite for a thorough software test is the design of relevant test cases, since they determine the kind and scope and hence the quality of the test. The classification-tree method and the graphical editor CTE (classification-tree editor) support the systematic design of black-box test cases. The classification-tree method is an approach to partition testing which uses a descriptive tree-like notation and which is especially suited for automation. Method and tool have already been tried out successfully on actual examples in various divisions of the Daim ler-Benz Group. Since the CTE has been so well received in in-house practice, it is now transformed into a product version with a number of additional features found to be useful during the practical trials.

Test Case Design Using Classification Trees and the Classification-Tree Editor CTE

The development of embedded systems is an essential industrial activity whose importance is increasing. Commonly, embedded software systems have to fulfil real-time requirements. The most important analytical method to assure the quality of real-time systems is dynamic testing. Testing is the only method which examines the actual run-time behaviour of embedded software systems, based on an execution in the real application environment. Dynamic aspects like the duration of computations, the memory actually needed, or the synchronisation of parallel processes are of major importance for the correct function of real-time systems, and have to be tested. The most important prerequisite for a thorough test is the design of relevant test cases. The classification-tree method was developed to overcome the shortcomings of current functional testing practice. It turns functional test case design into a process comprising several structured and systematised parts making it easy to handle, to understand, and also to document. Nevertheless, the classification-tree method alone is not sufficient for a comprehensive test of embedded systems. It is not well-suited for an examination of temporal correctness which is additionally essential to real-time systems. Already very small systems show a large variety of different execution times. Therefore, a new approach which concentrates on the temporal system behaviour was developed to complement the systematic test with the classification-tree method. It is based on the use of genetic algorithms and has been successfully tried out. The idea is to use genetic algorithms to search for input situations with very long or very short execution times, and to check whether they produce a temporal error. If all the times found correspond with the timing constraints, confidence in the temporal correctness is substantiated. Otherwise the real-time system has failed and an error is detected.

Systematic Testing of Real-Time Systems

Steigende Softwareumfange und die damit einhergehende Komplexitatssteigerung erfordern neue Herangehensweisen an die Entwicklung eingebetteter Fahrzeugsoftware. Als Reaktion auf diese Herausforderungen vollzieht sich seit einigen Jahren ein Paradigmenwechsel von der klassischen Programmentwicklung hin zu modellbasierten Techniken. Der Beitrag gibt einen Uberblick uber den Einsatz modellbasierter Entwicklungstechniken bei DaimlerChrysler und zeigt aktuelle Forschungsschwerpunkte auf.

/pdf/modellbasierte-entwicklung-eingebetteter-fahrzeugsoftware-sdi9h974l7.pdf

Matthias Grochtmann

Papers

Classification trees for partition testing

Verifying Timing Constraints of Real-Time Systems by Means of Evolutionary Testing

Test Case Design Using Classification Trees and the Classification-Tree Editor CTE

Systematic Testing of Real-Time Systems

Modellbasierte Entwicklung eingebetteter Fahrzeugsoftware bei DaimlerChrysler