Mutation Testing is a fault-based software testing technique that has been widely studied for over three decades The literature on Mutation Testing has contributed a set of approaches, tools, developments, and empirical results This paper provides a comprehensive analysis and survey of Mutation Testing The paper also presents the results of several development trend analyses These analyses provide evidence that Mutation Testing techniques and tools are reaching a state of maturity and applicability, while the topic of Mutation Testing itself is the subject of increasing interest

An Analysis and Survey of the Development of Mutation Testing

コンピュータ・サイエンス : ACM computing surveys

Testing involves examining the behaviour of a system in order to discover potential faults. Given an input for a system, the challenge of distinguishing the corresponding desired, correct behaviour from potentially incorrect behavior is called the “test oracle problem”. Test oracle automation is important to remove a current bottleneck that inhibits greater overall test automation. Without test oracle automation, the human has to determine whether observed behaviour is correct. The literature on test oracles has introduced techniques for oracle automation, including modelling, specifications, contract-driven development and metamorphic testing. When none of these is completely adequate, the final source of test oracle information remains the human, who may be aware of informal specifications, expectations, norms and domain specific information that provide informal oracle guidance. All forms of test oracles, even the humble human, involve challenges of reducing cost and increasing benefit. This paper provides a comprehensive survey of current approaches to the test oracle problem and an analysis of trends in this important area of software testing research and practice.

http://www0.cs.ucl.ac.uk/staff/m.harman/tse-oracle.pdf

The Oracle Problem in Software Testing: A Survey

Model-based testing (MBT) relies on models of a system under test and/or its environment to derive test cases for the system. This paper discusses the process of MBT and defines a taxonomy that covers the key aspects of MBT approaches. It is intended to help with understanding the characteristics, similarities and differences of those approaches, and with classifying the approach used in a particular MBT tool. To illustrate the taxonomy, a description of how three different examples of MBT tools fit into the taxonomy is provided. Copyright © 2011 John Wiley & Sons, Ltd.

/pdf/a-taxonomy-of-model-based-testing-approaches-242q024wa8.pdf

A taxonomy of model-based testing approaches

A Protocol Test Generation Procedure.

Formal methods and testing are two important approaches that assist in the development of high-quality software. While traditionally these approaches have been seen as rivals, in recent years a new consensus has developed in which they are seen as complementary. This article reviews the state of the art regarding ways in which the presence of a formal specification can be used to assist testing.

/pdf/using-formal-specifications-to-support-testing-28e8wig7se.pdf

Using formal specifications to support testing

/pdf/using-formal-methods-to-support-testing-2b67flflmy.pdf

Using formal methods to support testing

Fault-based testing of software checks the software implementation for a set of faults. Two previous papers on fault-based testing [Kuhn 1999; Tsuchiya and Kikuno 2002] represent the required behavior of the software as a Boolean specification represented in Disjunctive Normal Form (DNF) and then show that faults may be organized in a hierarchy. This article extends these results by identifying necessary and sufficient conditions for fault-based testing. Unlike previous solutions, the formal analysis used to derive these conditions imposes no restrictions (such as DNF) on the form of the Boolean specification.

Test conditions for fault classes in Boolean specifications

Given a test criterion, the number of test-sets satisfying the criterion may be very large, with varying fault detection effectiveness. This paper presents an experimental evaluation of the variation in fault detection effectiveness of all the test-sets for a given control-flow test criterion and a Boolean specification. The exhaustive experimental approach complements the earlier empirical studies that adopted analysis of some test-sets using random selection techniques. Three industrially used control-flow testing criteria, decision coverage (DC), full predicate coverage (FPC) and modified condition/decision coverage (MC/DC) have been analysed against four types of faults. The Boolean specifications used were taken from a past research paper and also generated randomly. To ensure that it is the test-set size, a variation of DC, decision coverage/random (DC/R), has also been considered against FPC and MC/DC criteria. In addition, a further analysis of variation in average effectiveness with respect to number of conditions in the decision has been done. The empirical results show that the MC/DC criterion is more reliable and stable in comparison to DC, DC/R and FPC.

Experimental evaluation of the variation in effectiveness for DC, FPC and MC/DC test criteria

Effectiveness of testing criteria is the ability to detect failure in a software program. We consider not only effectiveness of some testing criterion in itself but a variance of effectiveness of different test sets satisfied the same testing criterion. We name this property "tolerance" of a testing criterion and show that, for practical using a criterion, a high tolerance is as well important as high effectiveness. The results of empirical evaluation of tolerance for different criteria, types of faults and decisions are presented. As well as quite simple and well-known control-flow criteria, we study more complicated criteria: full predicate coverage, modified condition/decision coverage and reinforced condition/decision coverage criteria.

/pdf/tolerance-of-control-flow-testing-criteria-4wpiuxbgth.pdf

Kalpesh Kapoor

Papers

Using formal specifications to support testing

Using formal methods to support testing

Test conditions for fault classes in Boolean specifications

Experimental evaluation of the variation in effectiveness for DC, FPC and MC/DC test criteria

Tolerance of control-flow testing criteria