Jan Feenstra

Bio: Jan Feenstra is an academic researcher from University of Twente. The author has contributed to research in topics: Conformance testing & Automation. The author has an hindex of 5, co-authored 6 publications receiving 220 citations.

Formal Test Automation: A Simple Experiment

Abstract: In this paper we study the automation of test derivation and execution in the area of conformance testing. The test scenarios are derived from multiple specification languages: LOTOS, Promela and SDL. A central theme of this study is the usability of batch-oriented and on-the-fly testing approaches. To facilitate the derivation from multiple formal description techniques and the different test execution approaches, an open, generic environment called TorX is introduced. TorX enables plugging in existing or dedicated tools. We have carried out several experiments in testing a conference protocol, resulting in requirements on automated testing and benchmarking criteria.

Formal Test Automation: The Conference Protocol with PHACT

Abstract: We discuss a case study of automatic test generation and test execution based on formal methods The case is the Conference Protocol, a simple, chatbox-like protocol, for which (formal) specifications and multiple implementations are publicly available and which is also used in other case study experiments The tool used for test generation and test execution is Phact, the PHilips Automated Conformance Tester The formal method is (Extented) Finite State Machines which is the input language for Phact The experiment consists of developing a Finite State Machine specification for the Conference Protocol, generating 82 tests in TTCN with Phact, and executing these tests against 28 different implementations of the Conference Protocol, both correct and erroneous ones The result is that some erroneous implementations are not detected by the test cases These results are analysed, the merits of Extented Finite State Machines for specification are discussed, and the achievements of Phact are assessed Moreover, the results are compared with a previous experiment in which the same 28 implementations were tested based on specifications in LOTOS and Promela

Automated Testing in Practice: The Highway Tolling System

Abstract: In this paper we study the application of automated test derivation and execution based on formal specifications. The object of testing is the Payment Box (PB) of the Highway Tolling System, a device which handles electronic payments. Challenges of testing the PB are the transaction speed, parallelism and encryption. We describe a methodology for automated testing and apply this methodology to test the PB. We conclude that automation of the test process is feasible and beneficial, and evaluate our techniques, theory and tools for automated testing.

Specification based formal testing: the EasyLink case study

Abstract: Testing is, in most cases, a manual activity that is time consuming and error prone Automation, however, can severely reduce the associated costs In the project Cote de Resyste (COnformance TEsting of REactive SYSTEms) theory is being developed and a protoype tool is being built to support the automatic test generation and execution from formal specifications for reactive systems such as communication protocols and embedded system software [14] Industrial case studies are carried out by industrial partners to evaluate the theory and tools in practice, to identify potential bottlenecks and to suggest improvements In this paper we describe our experiences with one of such case studies, the EasyLink case study In this case study Audio/Video (AV) devices are automatically tested for compliance to the EasyLink standard The purpose of this case study is to check the viability of the theory and the tools developed in project by applying them in an industrial setting This paper discusses how the case study led to theoretical developments and tool improvements, and in that way provided valuable feedback that was used to guide the future direction of the project

Côte de Resyste in Progress

Abstract: Traditional (manual) testing of software systems is a costly, laborious and error-prone activity. It gets even more complicated nowadays with complex reactive software like embedded system software and communication protocols. Such software is characterized by a high degree of interactivity and concurrency. Cˆote de Resyste aims at developing methods and an integrated tool environment to support and, whenever possible, automate the testing process of reactive systems. Contrary to other test tools, this tool environment builds on a sound and well-defined theoretical basis. First results are the implementation of the prototype test tool TorX and its successful application to the Philips’ A/V Link protocol and to the academic Conference protocol case study. These experiments show that test automation, based on formal methods, is feasible and beneficial. Current work concentrates on improving TorX, on developing methods for effective selection of test sets and on generic test execution environments. A major part of the project is devoted to industrial case studies which are executed in close cooperation with Philips, Lucent Technologies and Interpay.

Model-based testing in practice

Abstract: Model-based testing is a new and evolving technique for generating a suite of test cases from requirements. Testers using this approach concentrate on a data model and generation infrastructure instead of hand-crafting individual tests. Several relatively small studies have demonstrated how combinatorial test generation techniques allow testers to achieve broad coverage of the input domain with a small number of tests. We have conducted several relatively large projects in which we applied these techniques to systems with millions of lines of code. Given the complexity of testing, the model-based testing approach was used in conjunction with test automation harnesses. Since no large empirical study has been conducted to measure efficacy of this new approach, we report on our experience with developing tools and methods in support of model-based testing. The four case studies presented here offer details and results of applying combinatorial test-generation techniques on a large scale to diverse applications. Based on the four projects, we offer our insights into what works in practice and our thoughts about obstacles to transferring this technology into testing organizations.

Model Based Testing with Labelled Transition Systems

Abstract: Model based testing is one of the promising technologies to meet the challenges imposed on software testing. In model based testing an implementation under test is tested for compliance with a model that describes the required behaviour of the implementation. This tutorial chapter describes a model based testing theory where models are expressed as labelled transition systems, and compliance is defined with the 'ioco' implementation relation. The ioco-testing theory, on the one hand, provides a sound and well-defined foundation for labelled transition system testing, having its roots in the theoretical area of testing equivalences and refusal testing. On the other hand, it has proved to be a practical basis for several model based test generation tools and applications. Definitions, underlying assumptions, an algorithm, properties, and several examples of the ioco-testing theory are discussed, involving specifications, implementations, tests, the ioco implementation relation and some of its variants, a test generation algorithm, and the soundness and exhaustiveness of this algorithm.

Model based testing with labelled transition systems

Abstract: Model based testing is one of the promising technologies to meet the challenges imposed on software testing. In model based testing an implementation under test is tested for compliance with a model that describes the required behaviour of the implementation. This tutorial chapter describes a model based testing theory where models are expressed as labelled transition systems, and compliance is defined with the 'ioco' implementation relation. The ioco-testing theory, on the one hand, provides a sound and well-defined foundation for labelled transition system testing, having its roots in the theoretical area of testing equivalences and refusal testing. On the other hand, it has proved to be a practical basis for several model based test generation tools and applications. Definitions, underlying assumptions, an algorithm, properties, and several examples of the ioco-testing theory are discussed, involving specifications, implementations, tests, the ioco implementation relation and some of its variants, a test generation algorithm, and the soundness and exhaustiveness of this algorithm.

CADP 2011: a toolbox for the construction and analysis of distributed processes

Abstract: CADP (Construction and Analysis of Distributed Processes) is a comprehensive software toolbox that implements the results of concurrency theory. Started in the mid 80s, CADP has been continuously developed by adding new tools and enhancing existing ones. Today, CADP benefits from a worldwide user community, both in academia and industry. This paper presents the latest release, CADP 2011, which is the result of a considerable development effort spanning the last five years. The paper first describes the theoretical principles and the modular architecture of CADP, which has inspired several other recent model checkers. The paper then reviews the main features of CADP 2011, including compilers for various formal specification languages, equivalence checkers, model checkers, compositional verification tools, performance evaluation tools, and parallel verification tools running on clusters and grids. Finally, the paper surveys some significant case studies.

Testing Concurrent Systems: A Formal Approach

Abstract: This paper discusses the use of formal methods in testing of concurrent systems. It is argued that formal methods and testing can be mutually profitable and useful. A framework for testing based on formal specifications is presented. This framework is elaborated for labelled transition systems, providing formal definitions of conformance, test execution and test derivation. A test derivation algorithm is given and its tool implementation is briefly discussed.