Mutation Testing is a fault-based software testing technique that has been widely studied for over three decades The literature on Mutation Testing has contributed a set of approaches, tools, developments, and empirical results This paper provides a comprehensive analysis and survey of Mutation Testing The paper also presents the results of several development trend analyses These analyses provide evidence that Mutation Testing techniques and tools are reaching a state of maturity and applicability, while the topic of Mutation Testing itself is the subject of increasing interest

An Analysis and Survey of the Development of Mutation Testing

We apply a model checker to the problem of test generation using a new application of mutation analysis. We define syntactic operators, each of which produces a slight variation on a given model. The operators define a form of mutation analysis at the level of the model checker specification. A model checker generates countersamples which distinguish the variations from the original specification. The countersamples can easily be turned into complete test cases, that is, with inputs and expected results. We define two classes of operators: those that produce test cases from which a correct implementation must differ, and those that produce test cases with which it must agree. There are substantial advantages to combining a model checker with mutation analysis. First, test case generation is automatic; each countersample is a complete test case. Second, in sharp contrast to program-based mutation analysis, equivalent mutant identification is also automatic. We apply our method to an example specification and evaluate the resulting test sets with coverage metrics on a Java implementation.

Using Model Checking to Generate Tests from Specifications | NIST

Mutation testing realizes the idea of using artificial defects to support testing activities. Mutation is typically used as a way to evaluate the adequacy of test suites, to guide the generation of test cases, and to support experimentation. Mutation has reached a maturity phase and gradually gains popularity both in academia and in industry. This chapter presents a survey of recent advances, over the past decade, related to the fundamental problems of mutation testing and sets out the challenges and open problems for the future development of the method. It also collects advices on best practices related to the use of mutation in empirical studies of software testing. Thus, giving the reader a “mini-handbook”-style roadmap for the application of mutation testing as experimental methodology.

/pdf/mutation-testing-advances-an-analysis-and-survey-2wtvd7tjwe.pdf

Mutation Testing Advances: An Analysis and Survey

This paper introduces a new paradigm for Mutation Testing, which we call Higher Order Mutation Testing (HOM Testing). Traditional Mutation Testing considers only first order mutants, created by the injection of a single fault. Often these first order mutants denote trivial faults that are easily killed. Higher order mutants are created by the insertion of two or more faults. The paper introduces the concept of a subsuming HOM; one that is harder to kill than the first order mutants from which it is constructed. By definition, subsuming HOMs denote subtle fault combinations. The paper reports the results of an empirical study of HOM Testing using 10 programs, including several non-trivial real-world subjects for which test suites are available.

/pdf/higher-order-mutation-testing-34frypiz67.pdf

Higher Order Mutation Testing

Assume-guarantee reasoning has long been advertised as an important method for decomposing proof obligations in system verification. Refinement mappings (homomorphisms) have long been advertised as an important method for solving the language-inclusion problem in practice. When confronted with large verification problems, we therefore attempted to make use of both techniques. We soon found that rather than offering instant solutions, the success of assume-guarantee reasoning depends critically on the construction of suitable abstraction modules, and the success of refinement checking depends critically on the construction of suitable witness modules. Moreover, as abstractions need to be witnessed, and witnesses abstracted, the process must be iterated. We present here the main lessons we learned from our experiments, in form of a systematic and structured discipline for the compositional verification of reactive modules. An infrastructure to support this discipline, and automate parts of the verification, has been implemented in the tool MOCHA.

You assume, we guarantee : Methodology and case studies

A set of mutation operators for SQL queries that retrieve information from a database is developed and tested against a set of queries drawn from the NIST SQL Conformance Test Suite. The mutation operators cover a wide spectrum of SQL features, including the handling of null values. Additional experiments are performed to explore whether the cost of executing mutants can be reduced using selective mutation or the test suite size can be reduced by using an appropriate ordering of the mutants. The SQL mutation approach can be helpful in assessing the adequacy of database test cases and their development, and as a tool for systematically injecting faults in order to compare different database testing techniques.

/pdf/mutating-database-queries-3j4aodyzmd.pdf

Mutating database queries

Generating test cases for compositions of web services is complex, due to their distributed nature and asynchronous behaviour. In this paper, a formal verification tool ‐ the SPIN model checker ‐ is used to generate test suite specifications for compositions specified in BPEL. A transition coverage criterion is employed to defin e a systematic procedure to select the test cases. The approach is applied to the “loan ap proval” sample composition.

Generating Test Cases Specifications for BPEL Compositions of Web Services Using SPIN

Populating test databases with meaningful test data is a difficult task as it involves generating data for many joined tables that must be diverse enough to be able to reveal faults and small enough to make the testing process efficient. This paper proposes an approach for the automatic generation of a test database for a set of SQL queries using a test criterion specifically tailored for the SQL language (SQLFpc). Given as input a schema database and a set of test requirements derived from the application of the test criterion to the target queries, the approach returns a database instance which satisfies the test requirements. Both the schema and the test requirements are modeled in the Alloy language, after which the analyzer generates the test database. The approach is evaluated on a real case study and the results show its feasibility, generating a test database of reduced size with an elevated coverage and mutation score.

http://giis.uniovi.es/testing/papers/ast-2010-qagrow.pdf

Constraint-based test database generation for SQL queries

In the field of database applications a considerable part of the business logic is implemented using a semi-declarative language: the Structured Query Language (SQL) Because of the different semantics of SQL compared with other procedural languages, the conventional coverage criteria for testing are not directly applicable This paper presents a criterion specifically tailored for SQL queries (SQLFpc) It is based on Masking Modified Condition Decision Coverage (MCDC) or Full Predicate Coverage and takes into account a wide range of the syntax and semantics of SQL, including selection, joining, grouping, aggregations, subqueries, case expressions and null values The criterion assesses the coverage of the test data in relation to the query that is executed and it is expressed as a set of rules that are automatically generated and efficiently evaluated against a test database The use of the criterion is illustrated in a case study, which includes complex queries Copyright © 2010 John Wiley & Sons, Ltd

This paper presents the SQLFpc coverage criterion specifically tailored for SQL queries The criterion is based on MCDC and measures the coverage of the test data in relation to the query that is executed It is expressed as a set of coverage rules that is automatically generated and efficiently evaluated against a test database A set of tools to automate the evaluation of the coverage is available at  Copyright © 2010 John Wiley & Sons, Ltd

This article was published online on [15 January 2010] An error was subsequently identified This notice is included in the online and print versions to indicate that both have been corrected [18 March 2010]

/pdf/full-predicate-coverage-for-testing-sql-database-queries-4jzj8o04q2.pdf

Full predicate coverage for testing SQL database queries

Functional testing of applications that process the information stored in databases often requires a careful design of the test database. The larger the test database, the more difficult it is to develop and maintain tests as well as to load and reset the test data. This paper presents an approach to reduce a database with respect to a set of SQL queries and a coverage criterion. The reduction procedures search the rows in the initial database that contribute to the coverage in order to find a representative subset that satisfies the same coverage as the initial database. The approach is automated and efficiently executed against large databases and complex queries. The evaluation is carried out over two real life applications and a well-known database benchmark. The results show a very large degree of reduction as well as scalability in relation to the size of the initial database and the time needed to perform the reduction.

Claudio de la Riva

Papers

Mutating database queries

Generating Test Cases Specifications for BPEL Compositions of Web Services Using SPIN

Constraint-based test database generation for SQL queries

Full predicate coverage for testing SQL database queries

Coverage-Aware Test Database Reduction