Regression testing is a testing activity that is performed to provide confidence that changes do not harm the existing behaviour of the software. Test suites tend to grow in size as software evolves, often making it too costly to execute entire test suites. A number of different approaches have been studied to maximize the value of the accrued test suite: minimization, selection and prioritization. Test suite minimization seeks to eliminate redundant test cases in order to reduce the number of tests to run. Test case selection seeks to identify the test cases that are relevant to some set of recent changes. Test case prioritization seeks to order test cases in such a way that early fault detection is maximized. This paper surveys each area of minimization, selection and prioritization technique and discusses open problems and potential directions for future research. Copyright © 2010 John Wiley & Sons, Ltd.

Regression testing minimization, selection and prioritization: a survey

With advanced computer technology, systems are getting larger to fulfill more complicated tasks: however, they are also becoming less reliable. Consequently, testing is an indispensable part of system design and implementation; yet it has proved to be a formidable task for complex systems. This motivates the study of testing finite stare machines to ensure the correct functioning of systems and to discover aspects of their behavior. A finite state machine contains a finite number of states and produces outputs on state transitions after receiving inputs. Finite state machines are widely used to model systems in diverse areas, including sequential circuits, certain types of programs, and, more recently, communication protocols. In a testing problem we have a machine about which we lack some information; we would like to deduce this information by providing a sequence of inputs to the machine and observing the outputs produced. Because of its practical importance and theoretical interest, the problem of testing finite state machines has been studied in different areas and at various times. The earliest published literature on this topic dates back to the 1950's. Activities in the 1960's mid early 1970's were motivated mainly by automata theory and sequential circuit testing. The area seemed to have mostly died down until a few years ago when the testing problem was resurrected and is now being studied anew due to its applications to conformance testing of communication protocols. While some old problems which had been open for decades were resolved recently, new concepts and more intriguing problems from new applications emerge. We review the fundamental problems in testing finite state machines and techniques for solving these problems, tracing progress in the area from its inception to the present and the stare of the art. In addition, we discuss extensions of finite state machines and some other topics related to testing.

Principles and methods of testing finite state machines-a survey

Exhaustive testing of computer software is intractable, but empirical studies of software failures suggest that testing can in some cases be effectively exhaustive. We show that software failures in a variety of domains were caused by combinations of relatively few conditions. These results have important implications for testing. If all faults in a system can be triggered by a combination of n or fewer parameters, then testing all n-tuples of parameters is effectively equivalent to exhaustive testing, if software behavior is not dependent on complex event sequences and variables have a small set of discrete values.

/pdf/software-fault-interactions-and-implications-for-software-2ckmz4q3dc.pdf

Software fault interactions and implications for software testing

Combinatorial Testing (CT) can detect failures triggered by interactions of parameters in the Software Under Test (SUT) with a covering array test suite generated by some sampling mechanisms. It has been an active field of research in the last twenty years. This article aims to review previous work on CT, highlights the evolution of CT, and identifies important issues, methods, and applications of CT, with the goal of supporting and directing future practice and research in this area. First, we present the basic concepts and notations of CT. Second, we classify the research on CT into the following categories: modeling for CT, test suite generation, constraints, failure diagnosis, prioritization, metric, evaluation, testing procedure and the application of CT. For each of the categories, we survey the motivation, key issues, solutions, and the current state of research. Then, we review the contribution from different research groups, and present the growing trend of CT research. Finally, we recommend directions for future CT research, including: (1) modeling for CT, (2) improving the existing test suite generation algorithm, (3) improving analysis of testing result, (4) exploring the application of CT to different levels of testing and additional types of systems, (5) conducting more empirical studies to fully understand limitations and strengths of CT, and (6) combining CT with other testing techniques.

http://romisatriawahono.net/lecture/rm/survey/software%20engineering/Software%20Testing/Nie%20-%20Combinatorial%20Testing%20-%202012.pdf

A survey of combinatorial testing

Computer communications

Regression testing is essential to ensure software quality. The test team applies a regression test suite to ensure that new or modified features do not regress (make worse) existing features. Although existing research has addressed many problems and put forward solutions, most regression test techniques are code-based. Code-based regression test selection is good for unit testing, but it has a scalability problem. When the size of the subject under test grows, it becomes hard to manage all the information and to create corresponding traceability matrices. In this paper, we describe a specification-based method for regression test selection.The basic model we use for describing requirements based on customer features or behaviors is the activity diagram, which is a notation of the Unified Modeling Language (UML). A process for identifying the affected test cases is presented. To summarize our approach, we select two kinds of regression tests: i) Targeted Tests, which ensure that important current customer features are still supported adequately in the new release and ii) Safety Tests, which are risk-directed, and ensure that potential problem areas are properly handled. Our test selection technique will be based on a practical risk analysis model.

Specification-based regression test selection with risk analysis

A regression test suite (RTS) is constructed to ensure that the changed parts of the system under test (SUT) behave as desired and that the unchanged parts of the SUT are not adversely affected. Model-based testing is a system testing technique in which systems are modeled by formal description languages, e.g., Extended Finite State Machine (EFSM) models. In this paper, a model-based RTS generation method based on EFSM dependence analysis is proposed. Twelve types of dependences are identified related to three types of elementary modifications (EMs), i.e., adding, deleting, and changing transitions in an EFSM model representing an SUT. These dependences capture the effects of the model on the EMs, the effects of the EMs on the model, and the side-effects of the EMs. The proposed method constructs an RTS by covering all occurrences of these dependences caused in a given EFSM model by a given set of EMs.

Model-based regression test suite generation using dependence analysis

A trend in software development is to assemble a system from a number of components. In many cases, the system is expected to function for multiple configurations of interchangeable components, leading to the problem of determining a set of system test configurations to fit a reasonable budget. One approach is to test all configurations that cover, for example, all two-way interactions. We investigate the feasibility of using an integer programming approach to solve the interaction test coverage problem exactly. We also examine the formulation to see if it provides insight into the NP-completeness of the interaction problem.

/pdf/formulation-of-the-interaction-test-coverage-problem-as-an-2686fh3ycs.pdf

Formulation of the Interaction Test Coverage Problem as an Integer Program

A model-based regression test suite (RTS) reduction method based on Extended Finite State Machine (EFSM) dependence analysis is proposed. Given an EFSM representing the requirements of a system under test (SUT) and a set of elementary modifications (EMs) on the EFSM, interaction patterns are identified related to each type of EMs, i.e., adding, deleting, and changing transitions in the EFSM. These interaction patterns capture the effects of the model on the EMs, the effects of the EMs on the model, and the side-effects of the EMs. The proposed method reduces the size of a given RTS by examining interaction patterns covered by each test case in the given RTS.

Robert L. Probert

Papers

Specification-based regression test selection with risk analysis

Model-based regression test suite generation using dependence analysis

Formulation of the Interaction Test Coverage Problem as an Integer Program

Regression test suite reduction using extended dependence analysis

A service-based method for the synthesis of communications protocols