Showing papers in "Software Testing, Verification & Reliability in 2004"
TL;DR: Some of the work undertaken in the use of metaheuristic search techniques for the automatic generation of test data is surveyed, discussing possible new future directions of research for each of its different individual areas.
Abstract: The use of metaheuristic search techniques for the automatic generation of test data has been a burgeoning interest for many researchers in recent years. Previous attempts to automate the test generation process have been limited, having been constrained by the size and complexity of software, and the basic fact that in general, test data generation is an undecidable problem. Metaheuristic search techniques oer much promise in regard to these problems. Metaheuristic search techniques are highlevel frameworks, which utilise heuristics to seek solutions for combinatorial problems at a reasonable computational cost. To date, metaheuristic search techniques have been applied to automate test data generation for structural and functional testing; the testing of grey-box properties, for example safety constraints; and also non-functional properties, such as worst-case execution time. This paper surveys some of the work undertaken in this eld, discussing possible new future directions of research for each of its dieren t individual areas.
1,351 citations
TL;DR: Some of the work undertaken in the use of metaheuristic search techniques for the automatic generation of test data is surveyed, discussing possible new future directions of research for each of its different individual areas.
Abstract: The use of metaheuristic search techniques for the automatic generation of test data has been a burgeoning interest for many researchers in recent years. Previous attempts to automate the test generation process have been limited, having been constrained by the size and complexity of software, and the basic fact that, in general, test data generation is an undecidable problem. Metaheuristic search techniques offer much promise in regard to these problems. Metaheuristic search techniques are high-level frameworks, which utilize heuristics to seek solutions for combinatorial problems at a reasonable computational cost. To date, metaheuristic search techniques have been applied to automate test data generation for structural and functional testing; the testing of grey-box properties, for example safety constraints; and also non-functional properties, such as worst-case execution time. This paper surveys some of the work undertaken in this field, discussing possible new future directions of research for each of its different individual areas. Copyright © 2004 John Wiley & Sons, Ltd.
181 citations
TL;DR: The use of metaheuristic search techniques for the automatic generation of test data has been a burgeoning interest for many researchers in recent years and previous attempts to automate the test gene selection process have failed.
Abstract: The use of metaheuristic search techniques for the automatic generation of test data has been a burgeoning interest for many researchers in recent years. Previous attempts to automate the test gene...
169 citations
TL;DR: AGENDA as mentioned in this paper is a set of tools to facilitate the use of this approach, including a parsing tool that gathers relevant information from the database schema and application, a tool that populates the database with meaningful data that satisfy database constraints, a test case generator that generates test cases for the application, and a tool to check the resulting database state after operations are performed by a database application.
Abstract: SUMMARY Database systems play an important role in nearly every modern organization, yet relatively little research effort has focused on how to test them. This paper discusses issues arising in testing database systems, presents an approach to testing database applications, and describes AGENDA, a set of tools to facilitate the use of this approach. In testing such applications, the state of the database before and after the user’s operation plays an important role, along with the user’s input and the system output. A framework for testing database applications is introduced. A complete tool set, based on this framework, has been prototyped. The components of this system are a parsing tool that gathers relevant information from the database schema and application, a tool that populates the database with meaningful data that satisfy database constraints, a tool that generates test cases for the application, a tool that checks the resulting database state after operations are performed by a database application, and a tool that assists the tester in checking the database application’s output. The design and implementation of each component of the system are discussed. The prototype described here is limited to applications consisting of a single SQL query. Copyright c � 2004 John Wiley & Sons, Ltd.
139 citations
TL;DR: Simulation‐based (stochastic) experiments, combined with optimized design‐of‐experiment plans, in the case study have shown a minimum productivity increase of 100 times in comparison to current practice without DoD STEP deployment.
Abstract: This paper presents some original solutions with regard to the deployment of the U.S. Department of Defense Simulation, Test and Evaluation Process (DoD STEP), using an automated target tracking radar system as a case study. Besides the integration of modelling and simulation, to form a model-based approach to the software testing process, the number of experiments, i.e. test cases, have been dramatically reduced by applying an optimized design-of-experiment plan and an orthogonal array-based robust testing methodology. Also, computer-based simulation at various abstraction levels of the system/software under test can serve as a test oracle. Simulation-based (stochastic) experiments, combined with optimized design-of-experiment plans, in the case study have shown a minimum productivity increase of 100 times in comparison to current practice without DoD STEP deployment. Copyright © 2004 John Wiley & Sons, Ltd.
45 citations
TL;DR: What is implied by risk‐based testing is examined, shows that its practice requires an understanding of risk, and points to the need for research into the topic and the development of a body of knowledge to underpin it.
Abstract: If software cannot be tested exhaustively, it must be tested selectively. But, on what should selection be based in order to maximize test effectiveness? It seems sensible to concentrate on the parts of the software where the risks are greatest, but what risks should be sought and how can they be identified and analysed? ‘Risk-based testing’ is a term in current use, for example in an accredited test-practitioner's course syllabus, but there is no broadly accepted definition of the phrase and no literature or body of knowledge to underpin the subject implied by it. Moreover, there has so far been no suggestion that it requires an understanding of the subject of risk. This paper examines what is implied by risk-based testing, shows that its practice requires an understanding of risk, and points to the need for research into the topic and the development of a body of knowledge to underpin it. Copyright © 2004 John Wiley & Sons, Ltd.
37 citations
TL;DR: This paper describes how BZ‐TT presents a unified view of these criteria to the validation engineer, and allows him or her to control the test case explosion on a coarse basis (choosing from a range of coverage criteria) as well as a fine basis (selecting options for each state or input variable).
Abstract: BZ-TESTING-TOOLS (BZ-TT) is a tool set for automated test case generation from B and Z specifications. BZ-TT uses boundary and cause–effect testing on the basis of the formal model. It has been used and validated on several industrial applications in the domain of critical software, particularly smart card and transport systems. This paper presents the test coverage criteria supported by BZ-TT. On the one hand, these correspond to various classical structural coverage criteria, but specialized to the case of B abstract machines. The paper gives algorithms for these in Prolog. On the other hand, BZ-TT introduces new coverage criteria for complex data structures, based on boundary analysis: this paper defines weak and strong state-boundary coverage, input-boundary coverage and output-boundary coverage. Finally, the paper describes how BZ-TT presents a unified view of these criteria to the validation engineer, and allows him or her to control the test case explosion on a coarse basis (choosing from a range of coverage criteria) as well as a fine basis (selecting options for each state or input variable). Copyright © 2004 John Wiley & Sons, Ltd.
35 citations
TL;DR: BZ‐Testing‐Tools is a tool‐set for automated model‐based test case generation from B abstract machines and Z specifications that uses boundary testing as well as cause–effect testing on the basis of the formal model.
Abstract: BZ-Testing-Tools (BZ-TT) is a tool-set for automated model-based test case generation from B abstract machines and Z specifications. BZ-TT uses boundary testing as well as cause–effect testing on the basis of the formal model. It has been used and validated on several industrial case studies in the domain of critical software: in particular for smart card applications and automotive embedded systems. The main idea of BZ-TT is to compute a boundary goal for each effect of the operations of the model and then to compute a preamble sequence of operations to place the system under test in such a state that satisfies the goal.
In this paper, the preamble computation search strategies used in BZ-TT are presented. More precisely, two algorithms based respectively on forward chaining and backward chaining are compared. These algorithms both use a customized set constraint solver, which is able to animate the formal model. These algorithms differ, however, in their capacity to reach the boundary goals efficiently. The results of applying the tools to an industrial windscreen wiper controller application are presented. Copyright © 2004 John Wiley & Sons, Ltd.
29 citations
TL;DR: What is implied by risk-based testing is examined, it is shown that its practice requires an understanding of risk, and the need for research into the topic and the development of a body of knowledge to underpin it is pointed to.
Abstract: If software cannot be tested exhaustively, it must be tested selectively. But, on what should selection be based in order to maximize test effectivenessq It seems sensible to concentrate on the parts of the software where the risks are greatest, but what risks should be sought and how can they be identified and analysedq ‘Risk-based testing’ is a term in current use, for example in an accredited test-practitioner's course syllabus, but there is no broadly accepted definition of the phrase and no literature or body of knowledge to underpin the subject implied by it. Moreover, there has so far been no suggestion that it requires an understanding of the subject of risk. This paper examines what is implied by risk-based testing, shows that its practice requires an understanding of risk, and points to the need for research into the topic and the development of a body of knowledge to underpin it. Copyright © 2004 John Wiley & Sons, Ltd.
28 citations
TL;DR: This paper describes how BZ-TT presents a unified view of these criteria to the validation engineer, and allows him or her to control the test case explosion on a coarse basis (choosing from a range of coverage criteria) as well as a fine basis (selecting options for each state or input variable).
Abstract: BZ-TESTING-TOOLS (BZ-TT) is a tool set for automated test case generation from B and Z specifications. BZ-TT uses boundary and cause–effect testing on the basis of the formal model. It has been used and validated on several industrial applications in the domain of critical software, particularly smart card and transport systems. This paper presents the test coverage criteria supported by BZ-TT. On the one hand, these correspond to various classical structural coverage criteria, but specialized to the case of B abstract machines. The paper gives algorithms for these in Prolog. On the other hand, BZ-TT introduces new coverage criteria for complex data structures, based on boundary analysis: this paper defines weak and strong state-boundary coverage, input-boundary coverage and output-boundary coverage. Finally, the paper describes how BZ-TT presents a unified view of these criteria to the validation engineer, and allows him or her to control the test case explosion on a coarse basis (choosing from a range of coverage criteria) as well as a fine basis (selecting options for each state or input variable). Copyright © 2004 John Wiley & Sons, Ltd.
18 citations
TL;DR: AGENDA as discussed by the authors is a set of tools to facilitate the use of this approach in testing database applications, including a parsing tool that gathers relevant information from the database schema and application, a tool that populates the database with meaningful data that satisfy database constraints, a test case generator that generates test cases for the application, and a tool to check the resulting database state after operations are performed by a database application.
Abstract: Database systems play an important role in nearly every modern organization, yet relatively little research effort has focused on how to test them This paper discusses issues arising in testing database systems, presents an approach to testing database applications, and describes AGENDA, a set of tools to facilitate the use of this approach In testing such applications, the state of the database before and after the user's operation plays an important role, along with the user's input and the system output A framework for testing database applications is introduced A complete tool set, based on this framework, has been prototyped The components of this system are a parsing tool that gathers relevant information from the database schema and application, a tool that populates the database with meaningful data that satisfy database constraints, a tool that generates test cases for the application, a tool that checks the resulting database state after operations are performed by a database application, and a tool that assists the tester in checking the database application's output The design and implementation of each component of the system are discussed The prototype described here is limited to applications consisting of a single SQL query Copyright © 2004 John Wiley & Sons, Ltd
TL;DR: Component users need to customize components they obtain from providers, because providers usually develop components for general use, so customization faults should be tested using both the black‐box part and the white-box part of a component.
Abstract: Component users need to customize components they obtain from providers, because providers usually develop components for general use. Although the customization is accomplished by modifying the interface of a component, faults from customization appear when the implementation part of a component and the interfaces interact. The implementation part is a black-box, whose source code is not available to a component user, while the interface is a white-box, whose source code is available for customization. Therefore, customization faults should be tested using both the black-box part and the white-box part of a component.
This paper proposes a new technique to test customization faults using software fault injection and mutation testing, and the technique is tailored to Enterprise JavaBeans. Test cases are selected by injecting faults not into the entire interface but into specific parts of the component's interface. The specific parts that are chosen control the effectiveness of the test cases. An empirical study to evaluate the technique is reported. Copyright © 2003 John Wiley & Sons, Ltd.
TL;DR: Investigation of the infection aspect of coverage criteria finds that a test set must execute the faulty statement, cause infection of the data state and then propagate the faulty data state to bring about a failure.
Abstract: Fault-detection effectiveness of coverage criteria has remained one of the controversial issues in recent years. In order to detect a fault, a test set must execute the faulty statement, cause infection of the data state and then propagate the faulty data state to bring about a failure. This paper sheds some light on the earlier contradictory results by investigating the infection aspect of coverage criteria.
For a given test criterion, the number of test sets satisfying the criterion may be very large, with varying fault-detection effectiveness. In a recent work the measure of variation in effectiveness of a test criterion was defined as ‘tolerance’. This paper presents an experimental evaluation of tolerance for control-flow test criteria by exhaustive test set generation, wherever possible. The approach used here is complementary to earlier empirical studies that adopted analysis of some test sets using random selection techniques. Four industrially used control-flow testing criteria, Condition Coverage (CC), Decision Condition Coverage (DCC), Full Predicate Coverage (FPC) and Modified Condition Decision Coverage (MCDC) have been analysed against four types of faults. A new test criterion, Reinforced Condition Decision Coverage (RCDC), is also analysed and compared. Copyright © 2004 John Wiley & Sons, Ltd.
TL;DR: The preamble computation search strategies used in BZ-TT are presented and two algorithms based respectively on forward chaining and backward chaining are compared, which use a customized set constraint solver to animate the formal model.
Abstract: BZ-Testing-Tools (BZ-TT) is a tool-set for automated model-based test case generation from B abstract machines and Z specifications. BZ-TT uses boundary testing as well as cause–effect testing on the basis of the formal model. It has been used and validated on several industrial case studies in the domain of critical software: in particular for smart card applications and automotive embedded systems. The main idea of BZ-TT is to compute a boundary goal for each effect of the operations of the model and then to compute a preamble sequence of operations to place the system under test in such a state that satisfies the goal.In this paper, the preamble computation search strategies used in BZ-TT are presented. More precisely, two algorithms based respectively on forward chaining and backward chaining are compared. These algorithms both use a customized set constraint solver, which is able to animate the formal model. These algorithms differ, however, in their capacity to reach the boundary goals efficiently. The results of applying the tools to an industrial windscreen wiper controller application are presented. Copyright © 2004 John Wiley & Sons, Ltd.A version of this paper was originally presented at SoftTest II: The Second U.K. Workshop on Software Testing Research, held at the University of York, U.K., 4–5 September 2003. It is reproduced here in modified form with the permission of the Workshop organizers
TL;DR: Simulation-based (stochastic) experiments, combined with optimized design-of-experiment plans, in the case study have shown a minimum productivity increase of 100 times in comparison to current practice without DoD STEP deployment.
Abstract: This paper presents some original solutions with regard to the deployment of the U.S. Department of Defense Simulation, Test and Evaluation Process (DoD STEP), using an automated target tracking radar system as a case study. Besides the integration of modelling and simulation, to form a model-based approach to the software testing process, the number of experiments, i.e. test cases, have been dramatically reduced by applying an optimized design-of-experiment plan and an orthogonal array-based robust testing methodology. Also, computer-based simulation at various abstraction levels of the system/software under test can serve as a test oracle. Simulation-based (stochastic) experiments, combined with optimized design-of-experiment plans, in the case study have shown a minimum productivity increase of 100 times in comparison to current practice without DoD STEP deployment. Copyright © 2004 John Wiley & Sons, Ltd.
TL;DR: This paper describes how a test set can be derived from a relatively big statechart and investigates the role of a few specific constraints in the reduction of test set size, depending on where in the system structure these constraints are used.
Abstract: An existing statechart testing method developed by the authors allows a tester to show the equivalence of the behaviour of an implementation of some software system to its Harel statechart specification, by testing. This requires a rather large test set and satisfaction of certain requirements for both a specification and an implementation. Introduction of additional constraints permits reduction in a test set size, without weakening of the conclusions obtained by testing. This paper describes how a test set can be derived from a relatively big statechart and investigates the role of a few specific constraints in the reduction of test set size, depending on where in the system structure these constraints are used. The main contribution of the paper is a demonstration that enforcing relatively weak constraints in the right place of a system's structure makes it possible to reduce the size of a test set by many orders of magnitude. The practical use of the contribution is to help developers structure a system under construction in such a way that it is completely testable using the described testing method with few additional constraints. Copyright © 2004 John Wiley & Sons, Ltd.
TL;DR: In this paper, the authors present a set of maxims that cover all the topics that the inspectors felt were important as they endeavoured to do good inspections and organize their comments into themes.
Abstract: Software inspections are an intensely people-centric activity. Even though this is routinely recognized in industry, much of the research focuses on inspection mechanics. During three years of inspection experiments, even though the main purpose of the experiments was to investigate the effectiveness of a particular technique, the inspectors involved provided broad comments on many other aspects of inspections. Their comments were collected and organized into themes. These themes are presented here as a set of maxims that cover all the topics that the inspectors felt were important as they endeavoured to do good inspections. Copyright © 2004 John Wiley & Sons, Ltd.
TL;DR: This paper describes how a test set can be derived from a relatively big statechart and investigates the role of a few specific constraints in the reduction of test set size, depending on where in the system structure these constraints are used.
Abstract: An existing statechart testing method developed by the authors allows a tester to show the equivalence of the behaviour of an implementation of some software system to its Harel statechart specification, by testing. This requires a rather large test set and satisfaction of certain requirements for both a specification and an implementation. Introduction of additional constraints permits reduction in a test set size, without weakening of the conclusions obtained by testing. This paper describes how a test set can be derived from a relatively big statechart and investigates the role of a few specific constraints in the reduction of test set size, depending on where in the system structure these constraints are used. The main contribution of the paper is a demonstration that enforcing relatively weak constraints in the right place of a system's structure makes it possible to reduce the size of a test set by many orders of magnitude. The practical use of the contribution is to help developers structure a system under construction in such a way that it is completely testable using the described testing method with few additional constraints. Copyright © 2004 John Wiley & Sons, Ltd.A version of this paper was originally presented at SoftTest II: The Second U.K. Workshop on Software Testing Research, held at the University of York, U.K., 4–5 September 2003. It is reproduced here in modified form with the permission of the Workshop organizers
TL;DR: An experimental evaluation of tolerance for control-flow test criteria by exhaustive test set generation, wherever possible, is presented, complementary to earlier empirical studies that adopted analysis of some test sets using random selection techniques.
Abstract: Fault-detection effectiveness of coverage criteria has remained one of the controversial issues in recent years. In order to detect a fault, a test set must execute the faulty statement, cause infection of the data state and then propagate the faulty data state to bring about a failure. This paper sheds some light on the earlier contradictory results by investigating the infection aspect of coverage criteria.For a given test criterion, the number of test sets satisfying the criterion may be very large, with varying fault-detection effectiveness. In a recent work the measure of variation in effectiveness of a test criterion was defined as ‘tolerance’. This paper presents an experimental evaluation of tolerance for control-flow test criteria by exhaustive test set generation, wherever possible. The approach used here is complementary to earlier empirical studies that adopted analysis of some test sets using random selection techniques. Four industrially used control-flow testing criteria, Condition Coverage (CC), Decision Condition Coverage (DCC), Full Predicate Coverage (FPC) and Modified Condition Decision Coverage (MCDC) have been analysed against four types of faults. A new test criterion, Reinforced Condition Decision Coverage (RCDC), is also analysed and compared. Copyright © 2004 John Wiley & Sons, Ltd.A version of this paper was originally presented at SoftTest II: The Second U.K. Workshop on Software Testing Research, held at the University of York, U.K., 4–5 September 2003. It is reproduced here in modified form with the permission of the Workshop organizers
TL;DR: This paper proposes a new technique to test customization faults using software fault injection and mutation testing, and the technique is tailored to Enterprise JavaBeans.
Abstract: Component users need to customize components they obtain from providers, because providers usually develop components for general use. Although the customization is accomplished by modifying the interface of a component, faults from customization appear when the implementation part of a component and the interfaces interact. The implementation part is a black-box, whose source code is not available to a component user, while the interface is a white-box, whose source code is available for customization. Therefore, customization faults should be tested using both the black-box part and the white-box part of a component.This paper proposes a new technique to test customization faults using software fault injection and mutation testing, and the technique is tailored to Enterprise JavaBeans. Test cases are selected by injecting faults not into the entire interface but into specific parts of the component's interface. The specific parts that are chosen control the effectiveness of the test cases. An empirical study to evaluate the technique is reported. Copyright © 2003 John Wiley & Sons, Ltd.