There is, I think, something ethereal about i —the square root of minus one. I remember first hearing about it at school. It seemed an odd beast at that time—an intruder hovering on the edge of reality.

Usually familiarity dulls this sense of the bizarre, but in the case of i it was the reverse: over the years the sense of its surreal nature intensified. It seemed that it was impossible to write mathematics that described the real world in …

I and i

A systematic literature review of techniques and metrics to reduce the cost of mutation testing

Model-based testing using UML activity diagrams: A systematic mapping study

This paper reports on results from a systematic review that characterizes the state-of-the-art on cost reduction for mutation testing. It analyzes the evolution of research on this topic and its underlying goals and techniques, and identifies metrics used to measure cost reduction. The mixed search strategy used automatic search, snowballing, and a survey of authors of primary studies. The analysis is based on a set of 165 peer-reviewed studies, from which 146 present either original or updated approaches and results for cost reduction of mutation testing. A list of 6 main goals for cost reduction is presented, and 22 techniques were identified. Historically, 18 metrics have been used to measure the gains and losses observed in experimental studies. In the last decade, substantial growth in the number of published studies was observed, particularly among techniques such as selective mutation, evolutionary algorithms, control-flow analysis, and higher order mutation.

A Systematic Review of Cost Reduction Techniques for Mutation Testing: Preliminary Results

Mutation testing research has often used the number of mutants as a surrogate measure for the true execution cost of generating and executing mutants. This poses a potential threat to the validity of the scientific findings reported in the literature. Out of 75 works surveyed in this paper, we found that 54 (72%) are vulnerable to this threat. To investigate the magnitude of the threat, we conducted an empirical evaluation using 10 real-world programs. The results reveal that: i) percentages of randomly sampled mutants differ from the true execution time, on average, by 44%, varying in difference from 19% to 91%; ii) errors arising from using the surrogate correlate with program size (ρ = 0.74) and number of mutants (ρ = 0.76), making the problem more pernicious for more realistic programs; iii) scientific findings concerning sampling strategies would have approximately 37% rank disagreement, indicating potentially dramatic impact on experiment validity. To investigate whether this threat matters in practice, we reproduced a seminal study on Selective Mutation (widely relied upon for more than two decades). The impact is stark: an inconclusive scientific finding using the surrogate is transformed to an unequivocal finding when using the true execution cost.

/pdf/cost-measures-matter-for-mutation-testing-study-validity-2s7wj108ud.pdf

Cost measures matter for mutation testing study validity

Nowadays, service oriented architecture (SOA) has become one mainstream paradigm for developing distributed applications. As the basic unit in SOA, web services can be composed to construct complex applications. The quality of web services and their compositions is critical to the success of SOA applications. Testing, as a major quality assurance technique, is confronted with new challenges in the context of service compositions. In this paper, we propose a scenario-oriented testing approach that can automatically generate test cases for service compositions. Our approach is particularly focused on the service compositions specified by Business Process Execution Language for web services (WS-BPEL), a widely recognized executable service composition language. In the approach, a WS-BPEL service composition is first abstracted into a graph model; test scenarios are then derived from the model; finally, test cases are generated according to different scenarios. We also developed a prototype tool implementing the proposed approach, and an empirical study was conducted to demonstrate the applicability and effectiveness of our approach. The experimental results show that the automatic scenario-oriented testing approach is effective in detecting many types of faults seeded in the service compositions.

/pdf/automated-testing-of-ws-bpel-service-compositions-a-scenario-3ojfnx6xgi.pdf

Automated Testing of WS-BPEL Service Compositions: A Scenario-Oriented Approach

Unified Modeling Language UML activity diagrams are widely used to model concurrent interaction among multiple objects In this paper, we propose a transformation-based approach to generating scenario-oriented test cases for applications modeled by UML activity diagrams Using a set of transformation rules, the proposed approach first transforms a UML activity diagram specification into an intermediate representation, from which it then constructs test scenarios with respect to the given concurrency coverage criteria The approach then finally derives a set of test cases for the constructed test scenarios The approach resolves the difficulties associated with fork and join concurrency in the UML activity diagram and enables control over the number of the resulting test cases We further implemented a tool to automate the proposed approach and studied its feasibility and effectiveness using a case study Experimental results show that the approach can generate test cases on demand to satisfy a given concurrency coverage criterion and can detect up to 765% of seeded faults when a weak coverage criterion is used With the approach, testers can not only schedule the software test process earlier, but can also better allocate the testing resources for testing concurrent applications Copyright © 2015John Wiley & Sons, Ltd

/pdf/a-transformation-based-approach-to-testing-concurrent-2vmxk8q9xr.pdf

A transformation-based approach to testing concurrent programs using UML activity diagrams

Nowadays, applications are increasingly deployed as Web services in the globally distributed cloud computing environment. Multiple services are normally composed to fulfill complex functionalities. Business Process Execution Language for Web Services (WS-BPEL) is an XML-based service composition language that is used to define a complex business process by orchestrating multiple services. Compared with traditional applications, WS-BPEL programs pose many new challenges to the quality assurance, especially testing, of service compositions. A number of techniques have been proposed for testing WS-BPEL programs, but only a few studies have been conducted to systematically evaluate the effectiveness of these techniques. Mutation testing has been widely acknowledged as not only a testing method in its own right but also a popular technique for measuring the fault-detection effectiveness of other testing methods. Several previous studies have proposed a family of mutation operators for generating mutants by seeding various faults into WS-BPEL programs. In this study, we conduct a series of empirical studies to evaluate the applicability and effectiveness of various mutation operators for WS-BPEL programs. The experimental results provide insightful and comprehensive guidance for mutation testing of WS-BPEL programs in practice. In particular, our work is the systematic study in the selection of effective mutation operators specifically for WS-BPEL programs.

/pdf/an-empirical-study-on-mutation-testing-of-ws-bpel-programs-5bv4evcq5i.pdf

Lin Pan

Papers

Automated Testing of WS-BPEL Service Compositions: A Scenario-Oriented Approach

A transformation-based approach to testing concurrent programs using UML activity diagrams

An Empirical Study on Mutation Testing of WS-BPEL Programs