Showing papers on "Test suite published in 1996"

Analyzing regression test selection techniques

Abstract: Regression testing is a necessary but expensive maintenance activity aimed at showing that code has not been adversely affected by changes. Regression test selection techniques reuse tests from an existing test suite to test a modified program. Many regression test selection techniques have been proposed, however, it is difficult to compare and evaluate these techniques because they have different goals. This paper outlines the issues relevant to regression test selection techniques, and uses these issues as the basis for a framework within which to evaluate the techniques. The paper illustrates the application of the framework by using it to evaluate existing regression test selection techniques. The evaluation reveals the strengths and weaknesses of existing techniques, and highlights some problems that future work in this area should address.

Test Generation with Inputs, Outputs and Repetitive Quiescence

Abstract: This paper studies testing based on labelled transition systems, using the assumption that implementations communicate with their environment via inputs and outputs. Such implementations are formalized by restricting the class of transition systems to those systems that can always accept input actions, as in Input/Output Automata. Implementation relations, formalizing the notion of correctness of these implementations with respect to labelled transition system specifications, are defined analogous to the theories of testing equivalence and preorder, and refusal testing. A test generation algorithm is given which is proved to produce a sound and exhaustive test suite from a specification, i.e., a test suite that fully characterizes the set of correct implementations.

The chaining approach for software test data generation

Abstract: Software testing is very labor intensive and expensive and accounts for a significant portion of software system development cost. If the testing process could be automated, the cost of developing software could be significantly reduced. Test data generation in program testing is the process of identifying a set of test data that satisfies a selected testing criterion, such as statement coverage and branch coverage. In this article we present a chaining approach for automated software test data generation which builds on the current theory of execution-oriented test data generation. In the chaining approach, test data are derived based on the actual execution of the program under test. For many programs, the execution of the selected statement may require prior execution of some other statements. The existing methods of test data generation may not efficiently generate test data for these types of programs because they only use control flow information of a program during the search process. The chaining approach uses data dependence analysis to guide the search process, i.e., data dependence analysis automatically identifies statements that affect the execution of the selected statement. The chaining approach uses these statements to form a sequence of statements that is to be executed prior to the execution of the selected statement. The experiments have shown that the chaining approach may significantly improve the chances of finding test data as compared to the existing methods of automated test data generation.

A framework for specification-based testing

Abstract: Test templates and a test template framework are introduced as useful concepts in specification-based testing. The framework can be defined using any model-based specification notation and used to derive tests from model-based specifications-in this paper, it is demonstrated using the Z notation. The framework formally defines test data sets and their relation to the operations in a specification and to other test data sets, providing structure to the testing process. Flexibility is preserved, so that many testing strategies can be used. Important application areas of the framework are discussed, including refinement of test data, regression testing, and test oracles.

Conformance testing with labelled transition systems: implementation relations and test generation

Abstract: This paper studies testing based on labelled transition systems, presenting two test generation algorithms with their corresponding implementation relations. The first algorithm assumes that implementations communicate with their environment via symmetric, synchronous interactions. It is based on the theory of testing equivalence and preorder, as is most of the testing theory for labelled transition systems, and it is found in the literature in some slightly different variations. The second algorithm is based on the assumption that implementations communicate with their environment via inputs and outputs. Such implementations are formalized by restricting the class of labelled transition systems to those systems that can always accept input actions. For these implementations a testing theory is developed, analogous to the theory of testing equivalence and preorder. It consists of implementation relations formalizing the notion of conformance of these implementations with respect to labelled transition system specifications, test cases and test suites, test execution, the notion of passing a test suite, and the test generation algorithm, which is proved to produce sound test suites for one of the implementation relations.

Using On-The-Fly Verification Techniques for the Generation of test Suites

Abstract: In this paper we attempt to demonstrate that on-the-fly techniques, developed in the context of verification, can help in deriving test suites Test purposes are used in practice to select test cases according to some properties of the specification We define a consistency pre-order linking test purposes and specifications We give a set of rules to check this consistency and to derive a complete test case with preamble, postamble, verdicts and timers The algorithm, which implements the construction rules, is based on a depth first traversal of a synchronous product between the test purpose and the specification We shortly relate our experience on an industrial protocol with TGV, a first prototype of the algorithm implemented as a component of the C ADP toolbox

An experimental evaluation of data flow and mutation testing

Abstract: Two experimental comparisons of data flow and mutation testing are presented. These techniques are widely considered to be effective for unit-level software testing, but can only be analytically compared to a limited extent. We compare the techniques by evaluating the effectiveness of test data developed for each. We develop ten independent sets of test data for a number of programs: five to satisfy the mutation criterion and five to satisfy the all-uses data-flow criterion. These test sets are developed using automated tools, in a manner consistent with the way a test engineer might be expected to generate test data in practice. We use these test sets in two separate experiments. First we measure the effectiveness of the test data that was developed for one technique in terms of the other. Second, we investigate the ability of the test sets to find faults. We place a number of faults into each of our subject programs, and measure the number of faults that are detected by the test sets. Our results indicate that while both techniques are effective, mutation-adequate test sets are closer to satisfying the data flow criterion, and detect more faults.

Test Generation with Inputs, Outputs, and Quiescence

Abstract: This paper studies testing based on labelled transition systems, using the assumption that implementations communicate with their environment via inputs and outputs. Such implementations are formalized by restricting the class of transition systems to those systems that can always accept input actions, as in input/output automata. Implementation relations, formalizing the notion of conformance of these implementations with respect to labelled transition system specifications, are defined analogous to the theory of testing equivalence and preorder. A test generation algorithm is given, which is proved to produce a sound and exhaustive test suite from a specification, i.e., a test suite that fully characterizes the set of correct implementations.

Method and system for testing hardware and/or software applications

Abstract: An adaptable system and method for testing a plurality of hardware and/or software applications. The system and method include a test case generator for generating a plurality of test cases, each test case corresponding to one of the plurality of applications, and a multi-layered interface for communicating between corresponding test cases and applications. The system and method also include means for executing the test cases to generate test data, and means for recording the test data generated during execution of the test cases.

Automated test data generation for programs with procedures

Abstract: Test data generation in program testing is the process of identifying a set of test data that satisfies a selected testing criterion, such as, statement coverage or branch coverage. The existing methods of test data generation are limited to unit testing and may not efficiently generate test data for programs with procedures. In this paper we present an approach for automated test data generation for programs with procedures. This approach builds on the current theory of execution-oriented test data generation. In this approach, test data are derived based on the actual execution of the program under test. For many programs, the execution of the selected statement may require prior execution of some other statements that may be part of some procedures. The existing methods use only control flow information of a program during the search process and may not efficiently generate test data for these types of programs because they are not able to identify statements that affect execution of the selected statement. Our approach uses data dependence analysis to guide the process of test data generation. Data dependence analysis automatically identifies statements (or procedures) that affect the execution of the selected statement and this information is used to guide the search process. The initial experiments have shown that this approach may improve the chances of finding test data.

Test definition tool

Abstract: In a diagnostic application, a plurality of independent test modules are executed in a multi-tasking fashion. The diagnostic application is modular with a front end module issuing commands to lower level modules. A lower level test dispatcher module receives information from the plurality of test modules pertaining to test parameters, including whether the test module is multitaskable. A test definition tool is provided to visually develop test scripts by moving icons from one list to another. The output of the test definition tool is a scripting language readable by the diagnostic application.

System and method for selecting test units to be re-run in software regression testing

Abstract: A system and method for selective regression testing of a software system that determines which test units of a test suite must be re-run after a modification to the software system. The entities which are covered by each test unit are identified. When the software system is modified the entities which were changed by the modification are identified. The test units which need to be re-run are determined by analyzing the change information and the coverage information to select those test units that cover changed entities.

Automatic software testing tool

Abstract: A method of testing a software module using a genetic algorithm to generate a best test script is provided. The software module has a number of possible states. A set of state machines is created which represent a definition for either a user interface or an application program interface (API) of the software module in terms of the possible states. From the state machines, a set of test cases is automatically generated, such that each test case consists of code for manipulating the user interface or API. A genetic algorithm creates populations of test scripts from the test cases, in which each test script includes a number of test cases. Each test script from each successive generation of test scripts is executed by applying the script as input to the software module. A code coverage analyzer provides a measure of code coverage to the genetic algorithm for each test script. The genetic algorithm uses the measure of code coverage as a fitness value in generating future populations of test scripts and in determining a best script.

Determination of software functionality

Abstract: A method of determining the functionality of a software system includes inputting a test plan to the software system via a software testing interface program, and logging an indication of one or more resulting outputs of the software system compared to expected output(s). The test plan invokes a sequence of test scripts and includes associated parameter inputs for the test scripts, and an expected output of the function or transaction of each test script. The test scripts are selected from a set of test scripts each able, when input to the software system via a software testing interface program, to prompt performance of a transaction or function for which the software system is designed.

Test support tool system and method

Abstract: A test support tool system and method produce software test programs from a logical description of selected software. Test programs are created by producing a cause-effect graph from the logical description, creating a decision table, producing test cases, and synthesizing test cases into a test program. The test support tool system includes an interface for receiving a logical description of software, a logical database, an analysis and test case generation module, a control module, and a test program synthesis module.

Program Slicing‐Based Regression Testing Techniques

Abstract: After changes are made to a previously tested program, a goal of regression testing is to perform retesting based on the modifications while maintaining the same testing coverage as completely retesting the program. This paper presents a novel approach to data flow based regression testing that uses slicing algorithms for the explicit detection of definition-use associations that are affected by a program change. An important benefit of this slicing technique is that, unlike previous techniques, neither data flow history nor recomputation of data flow for the entire program is required to detect affected definition-use associations. The program changes drive the recomputation of the required partial data flow through slicing. Another advantage is that the technique achieves the same testing coverage with respect to the affected definition-use associations as a complete retest of the program, without maintaining a test suite. Thus, the overhead of maintaining and updating a test suite is eliminated.

Method and system for generating a computer program test suite using dynamic symbolic execution

Abstract: A method and system for generating a test suite for a computer program. The computer program comprises program statements and program variables, including at least one input statement having one or more input variables, that are grouped into code blocks and stored in a program database. The test suite comprises sets of inputs. Each of the sets of inputs corresponds to each of the input statements. The program statements corresponding to a candidate code block are read from the program database. Each of the input variables for each input statement and each of the program variables are represented in symbolic form as a symbolic memory value and transforming each program statement dependent on such an input variable into a symbolic expression. A trial set of inputs for each of the input statements is created by finding a solution to the symbolic expression comprising actual input values corresponding to each symbolic memory value using dynamic symbolic exeuction. An execution run of the computer program is performed using the trial set of inputs and analyzing results obtained from the execution run for coverage of the candidate code block. The trial set of inputs are stored into the test suite if coverage of the candidate code block was obtained.

MDO Test Suite at NASA Langley Research Center

Abstract: NASA Langley Research Center supports a wide variety of multidisciplinary design optimization (MDO) research and requires a set of standard MDO test problems for evaluating and comparing the products of this research. This paper proposes a World-Wide-Web based test suite for collecting, distributing and maintaining the standard test problems. A prototype suite of 10 problems, including written problem descriptions, benchmark solution methods, sample input and output files, and source code is available from the Langley internet site. Here, design of the MDO test suite is discussed; typical test problems are described; and sample web pages are illustrated.

Coverage-Directed Test Generation Using Symbolic Techniques

Abstract: In this paper, we present a verification methodology that integrates formal verification techniques with verification by simulation, thereby providing means for generating simulation test suites that ensure coverage. We derive the test suites by means of BDD-based symbolic techniques for describing and traversing the implementation state space. In our approach, we provide a high-level of control over the generated test suites; a powerful abstraction mechanism directs the generation procedure to specific areas, that are the focus for verification, thereby withstanding the state explosion problem. The abstraction is achieved by partitioning the implementation state variables into categories of interest. We also depart from the traditional graph-algorithmic model for conformance testing; instead, using temporal logic assertions, we can generate a test suite where the set of state sequences (paths) satisfies some temporal properties as well as guaranteeing transition coverage. Our methodology has been successfully applied to the generation of test suites for IBM PowerPC and AS/400 systems.

On fault coverage of tests for finite state specifications

Abstract: Testing is a trade-off between increased confidence in the correctness of the implementation under test and constraints on the amount of time and effort that can be spent in testing. Therefore, the coverage, or adequacy of the test suite, becomes a very important issue. In this paper, we analyze basic ideas underlying the techniques for fault coverage analysis and assurance mainly developed in the context of protocol conformance testing based on finite state models. Special attention is paid to parameters which determine the testability of a given specification and influence the length of a test suite which guarantees complete fault coverage. We also point out certain issues which need further study.

Assertion-oriented automated test data generation

Abstract: Assertions are recognized as a powerful tool for automatic run time detection of software errors. However, existing testing methods do not use assertions to generate test cases. We present a novel approach of automated test data generation in which assertions are used to generate test cases. In this approach the goal is to identify test cases on which an assertion is violated. If such a test is found then this test uncovers an error in the program. The problem of finding program input on which an assertion is violated may be reduced to the problem of finding program input on which a selected statement is executed. As a result, the existing methods of automated test data generation for white box testing may be used to generate tests to violate assertions. The experiments have shown that this approach may significantly improve the chances of finding software errors as compared to the existing methods of test generation.

Method and computer program product for generating a computer program product test that includes an optimized set of computer program product test cases, and method for selecting same

Abstract: A method for selecting a set of test cases which may be used to test a software program product is disclosed. The program to be tested may have a number of code blocks that may be exercised during execution of the program. The method includes identifying each of the code blocks that may be exercised, and determining a time for executing each of the test cases in the set. A set of the test cases is then selected that exercises a maximum number of the identified code blocks that can be exercised in a minimum time. The selection step may be performed by executing a genetic algorithm for determining which subset of test cases to use, using a combination of time and coverage as a fitness value.

Software testing apparatus and method

Abstract: The present invention discloses a computer-implemented method for testing a software product using a server computer system and a plurality of servant computer systems which are networked together. The method includes the step of providing the server computer system with the software product to be tested and an associated test suite. The test suite being designed to exercise the software product and generate a test suite log indicative of test results obtained from executing the test suite. The server computer system then transmits the software product and the test suite to user-defined servant computer systems such that the software product and test suite is executed on each of the user-defined servant computer systems. A test suite log is then generated on each of the selected servant computer systems and transmitted back to the server computer system.

Object oriented framework for testing software

Abstract: A software testing system tests software. The software testing system includes a set-up and control system, and one or more test systems connected to the set-up and control system. The set-up and control system includes a user interface for interacting with a user of the software testing system. The set-up and control system stores test data and test configurations. Each test system sets up test cases from the test data and the test configurations stored in the set-up and control system. Each test case inherits attributes from at least one test object within the test system. The inherited attributes facilitates the interaction of the test system with the test cases.

Method and apparatus for software testing using a differential testing technique to test compilers

Abstract: Techniques used in testing software are described. A test generator produces a source program used to test two or more compilers using a differential testing technique. The test generator includes a grammar with added semantic constraints to minimize the generation of non-conforming source programs. The source program is a conforming source program conforming to constraints included in a programming language standard. By using properties of a conforming source code, a differential testing technique is described in which a test failure indicates that one or more of the compilers is not processing the source program correctly in accordance with the programming language standard. If a test failure is detected, the source program causing the test failure is reduced using various reduction and simplification techniques.

Structural specification-based testing with ADL

Abstract: This paper describes a specification-based black-box technique for testing program units. The main contribution is the method that we have developed to derive test conditions, which are descriptions of test cases, from the formal specification of each program unit. The derived test conditions are used to guide test selection and to measure comprehensiveness of existing test suites. Our technique complements traditional code-based techniques such as statement coverage and branch coverage. It allows the tester to quickly develop a black-box test suite.In particular, this paper presents techniques for deriving test conditions from specifications written in the Assertion Definition Language (ADL) [SH94], a predicate logic-based language that is used to describe the relationships between inputs and outputs of a program unit. Our technique is fully automatable, and we are currently implementing a tool based on the techniques presented in this paper.

Weighting for godot: learning heuristics for GSAT

Abstract: We investigate an improvement to GSAT which associates a weight with each clause. GSAT moves to assignments maximizing the weight of satisfied clauses and this weight is incremented when GSAT moves to an assignment in which this clause is unsatisfied. We present results showing that this algorithm and its variants outperform one of the best known modifications of GSAT to date using two metrics: number of solved problems on a single try, and minimum mean number of flips to solve a test suite of problems.

Global predicate analysis and its application to register allocation

Abstract: To fully utilize the wide machine resources in modern high-performance microprocessors it is necessary to exploit parallelism beyond individual basic blocks. Architectural support for predicated execution increases the degree of instruction level parallelism by allowing instructions from different basic blocks to be converted to straight-line code guarded by boolean predicates. However predicated execution also presents significant challenges to an optimizing compiler. For example, in live range analysis, a predicated definition does not necessarily end the live range of a virtual register. This paper describes techniques to analyze the relations among predicates in order to improve the precision and effectiveness of various compiler analysis and transformation phases in the presence of predicated code. Our predicate analysis operates globally to obtain relations among predicates. Moreover, we analyze control flow and predication in a single unified framework. The result can be queried by subsequent optimization and analysis phases. Based on this framework, we extend a traditional method to a predicate-aware register allocator which takes global predicate relations into account. We have implemented the proposed algorithms to effectively reduce register pressure. Our experimental results show 24.6% of a large test suite obtain, on average, 20.71% and better register allocation due to the algorithms presented in this paper.