Mutation Testing is a fault-based software testing technique that has been widely studied for over three decades The literature on Mutation Testing has contributed a set of approaches, tools, developments, and empirical results This paper provides a comprehensive analysis and survey of Mutation Testing The paper also presents the results of several development trend analyses These analyses provide evidence that Mutation Testing techniques and tools are reaching a state of maturity and applicability, while the topic of Mutation Testing itself is the subject of increasing interest

An Analysis and Survey of the Development of Mutation Testing

This paper introduces a new paradigm for Mutation Testing, which we call Higher Order Mutation Testing (HOM Testing). Traditional Mutation Testing considers only first order mutants, created by the injection of a single fault. Often these first order mutants denote trivial faults that are easily killed. Higher order mutants are created by the insertion of two or more faults. The paper introduces the concept of a subsuming HOM; one that is harder to kill than the first order mutants from which it is constructed. By definition, subsuming HOMs denote subtle fault combinations. The paper reports the results of an empirical study of HOM Testing using 10 programs, including several non-trivial real-world subjects for which test suites are available.

/pdf/higher-order-mutation-testing-34frypiz67.pdf

Higher Order Mutation Testing

This paper presents a learning-based approach to detecting failures in reactive systems. The technique is based on inferring models of multiple implementations of a common specification which are pair-wise cross-checked for equivalence. Any counterexample to equivalence is flagged as suspicious and has to be analysed manually. Hence, it is possible to find possible failures in a semi-automatic way without prior modelling. 
We show that the approach is effective by means of a case study. For this case study, we carried out experiments in which we learned models of five implementations of MQTT brokers/servers, a protocol used in the Internet of Things. Examining these models, we found several violations of the MQTT specification. All but one of the considered implementations showed faulty behaviour. In the analysis, we discuss effectiveness and also issues we faced.

Model-Based Testing IoT Communication via Active Automata Learning

This paper presents a learning-based approach to detecting failures in reactive systems The technique is based on inferring models of multiple implementations of a common specification which are pair-wise cross-checked for equivalence Any counterexample to equivalence is flagged as suspicious and has to be analysed manually Hence, it is possible to find possible failures in a semi-automatic way without prior modelling We show that the approach is effective by means of a case study For this case study, we carried out experiments in which we learned models of five implementations of MQTT brokers/servers, a protocol used in the Internet of Things Examining these models, we found several violations of the MQTT specification All but one of the considered implementations showed faulty behaviour In the analysis, we discuss effectiveness and also issues we faced

Many modern software systems are composed of multiple execution flows that run simultaneously, spanning from applications designed to exploit the power of modern multi-core architectures to distributed systems consisting of multiple components deployed on different physical nodes We collectively refer to such systems as concurrent systems  Concurrent systems are difficult to test, since the faults that derive from their concurrent nature depend on the interleavings of the actions performed by the individual execution flows Testing techniques that target these faults must take into account the concurrency aspects of the systems The increasingly rapid spread of parallel and distributed architectures led to a deluge of concurrent software systems, and the explosion of testing techniques for such systems in the last decade The current lack of a comprehensive classification, analysis and comparison of the many testing techniques for concurrent systems limits the understanding of the strengths and weaknesses of each approach and hampers the future advancements in the field This survey provides a framework to capture the key features of the available techniques to test concurrent software systems, identifies a set of classification criteria to review and compare the available techniques, and discusses in details their strengths and weaknesses, leading to a thorough assessment of the field and paving the road for future progresses

/pdf/a-survey-of-recent-trends-in-testing-concurrent-software-1r5imzzt1k.pdf

A Survey of Recent Trends in Testing Concurrent Software Systems

Fault-based testing is a technique where testers anticipate errors in a system under test in order to assess or generate test cases. The idea is to have enough test cases capable of detecting these anticipated errors. This paper presents a theory and technique for generating fault-based test cases for concurrent systems. The novel idea is to generate test purposes from faults that have been injected into a model of the system under test. Such test purposes form a specification of a more detailed test case that can detect the injected fault. The theory is based on the notion of refinement. The technique is automated using the TGV test case generator and an equivalence checker of the CADP tools. A case study of testing web servers demonstrates the practicability of the approach.

/pdf/from-faults-via-test-purposes-to-test-cases-on-the-fault-r12u2br1sl.pdf

Carlo Corrales Delgado

Papers

From faults via test purposes to test cases: on the fault-based testing of concurrent systems