Showing papers on "Test harness published in 2009"

Formal verification of practical MPI programs

Abstract: This paper considers the problem of formal verification of MPI programs operating under a fixed test harness for safety properties without building verification models. In our approach, we directly model-check the MPI/C source code, executing its interleavings with the help of a verification scheduler. Unfortunately, the total feasible number of interleavings is exponential, and impractical to examine even for our modest goals. Our earlier publications formalized and implemented a partial order reduction approach that avoided exploring equivalent interleavings, and presented a verification tool called ISP. This paper presents algorithmic and engineering innovations to ISP, including the use of OpenMP parallelization, that now enables it to handle practical MPI programs, including:(i)~ParMETIS - a widely used hypergraph partitioner, and (ii)~MADRE - a Memory Aware Data Re-distribution Engine, both developed outside our group. Over these benchmarks, ISP has automatically verified up to 14K lines of MPI/C code, producing error traces of deadlocks and assertion violations within seconds.

System evaluation apparatus

Abstract: An apparatus for evaluating a change of an existing system to a modified system by using a test system including a subsystem having the same configuration with either of the existing subsystem and the modified system includes: a memory for storing configuration information of the test system and test information; and a processor for executing a process including: activating the test system to execute test on the subsystem, collecting test information from the test system, and storing the test result information while the subsystem has the same configuration as the existing system; confirming the test system has been modified; and activating the test system to execute test on the subsystem, collecting test result information from the test system, and storing the test result information while the subsystem has the same configuration as the modified system.

UML Activity Diagram-Based Automatic Test Case Generation For Java Programs

Abstract: Test case generation based on design specifications is an important part of testing processes. In this paper, Unified Modeling Language activity diagrams are used as design specifications. By setting up several test adequacy criteria with respect to activity diagrams, an automatic approach is presented to generate test cases for Java programs. Instead of directly deriving test cases from activity diagrams, this approach selects test cases from a set of randomly generated ones according to a given test adequacy criterion. In the approach, we first instrument a Java program under testing according to its activity diagram model, and randomly generate abundant test cases for the program. Then, by running the instrumented program we obtain the corresponding program execution traces. Finally, by matching these traces with the behavior of the activity diagram, a reduced set of test cases are selected according to the given test adequacy criterion. This approach can also be used to check the consistency between the program execution traces and the behavior of activity diagrams.

Automated test execution plan generation

Abstract: A set of user-configured testing parameters for a software application under test can be received by a test execution plan generation tool. At least one testing objective for the software application can be defined by user-configurable testing parameters. A test execution plan can be automatically generated utilizing the user-configured testing parameters and a predefined test execution plan data model. The predefined test execution plan data model can include testing metadata representing software testing domain data for a software testing system being used to evaluate the software application under test. The testing metadata can specify details of finite shared resources of a test center shared by all projects under test. The generated test execution plan can detail specific allocation of the finite shared resources dedicated towards at least one phase of testing the software application to satisfy the at least one testing objective for the software application.

Systems and methods for visual test authoring and automation

Abstract: A method of a visual test authoring and automation solution framework for an enterprise comprising supporting the creation of a test case for a visual application. First, the framework allows a user to assign a user-defined name to a test element, and select an action to be performed on the test element from a menu of actions. Second, the framework stores a mapping of the user-defined name assigned to the test element to a coded name in a corresponding language of an automated testing tool. Lastly, the system uses the mapping and the action selected to create the test case in the corresponding language of the automated testing tool.

System and method of automated fault analysis and diagnostic testing of an aircraft

Abstract: A system for automated fault analysis and execution of post-flight or pre-flight tests for an aircraft is provided. The system includes a maintenance test client system onboard the aircraft, and a ground-based maintenance test server system that is remote from the aircraft. The maintenance test client system is configured to collect fault data for the aircraft, and to wirelessly transmit the fault data to the ground-based maintenance test server system after the aircraft has landed. The ground-based maintenance test server system is configured to receive the fault data, to generate a test checklist that is influenced by the fault data, and to transmit the test checklist to the maintenance test client system. The maintenance test client system wirelessly receives the test checklist, and initiates execution of at least one test specified in the test checklist.

Implementing an Effective Test Automation Framework

Abstract: Testing automation tools enable developers and/or testers to easily automate the entire process of testing in software development. Nevertheless, adopting automated testing is not easy and unsuccessful due to a lack of key information and skills. In order to help solve such problems, we have designed a new framework to support a clear overview of the test design and/or plan for automating tests in distributed environments. Those that are new to testing do not need to delve into complex automation tools or test scripts. This framework allows a programmer or tester to graphically specify the granularity of execution control and aids communication between various stakeholders and software developers. It also enables them to perform the automated Continuous Integration environment. In this paper, we describe details on experiences and usage of the proposed framework.

Reducing the Costs of Bounded-Exhaustive Testing

Abstract: Bounded-exhaustive testing is an automated testing methodology that checks the code under test for all inputs within given bounds: first the user describes a set of test inputs and provides test oracles that check test outputs; then the tool generates all the inputs, executes them on the code under test, and checks the outputs; and finally the user inspects failing tests to submit bug reports. The costs of bounded-exhaustive testing include machine time for test generation and execution (which translates into human time waiting for these results) and human time for inspection of results. This paper proposes three techniques that reduce these costs. Sparse Test Generation skips some tests to reduce the time to the first failing test. Structural Test Merging generates a smaller number of larger test inputs (rather than a larger number of smaller test inputs) to reduce test generation and execution time. Oracle-based Test Clustering groups failing tests to reduce the inspection time. Results obtained from the bounded-exhaustive testing of the Eclipse refactoring engine show that these three techniques can substantially reduce the costs while mostly preserving fault-detection capability.

Lifecycle management of automated testing

Abstract: Systems and methods for lifecycle management of automated testing are disclosed. In one embodiment, a method includes processing multiple manual test cases for an application under test, associating a set of reusable test scripts to the manual test cases, where the set of reusable test scripts is selected from a library of reusable test scripts, and executing the set of reusable test scripts for the application under test using an automated testing tool associated with the set of reusable test scripts.

Model-Based Testing of Web Applications Using NModel

Abstract: We show how model-based on-the-fly testing can be applied in the context of web applications using the NModel toolkit. The concrete case study is a commercial web-based positioning system called WorkForce Management (WFM) which interacts with a number of other services, such as billing and positioning, through a mobile operator. We describe the application and the testing, and discuss the test results.

Exploring the use of a test automation framework

Abstract: It is known that software testers, more often than not, lack the time needed to fully test the delivered software product within the time period allotted to them. When problems in the implementation phase of a development project occur, it normally causes the software delivery date to slide. As a result, testers either need to work longer hours, or supplementary resources need to be added to the test team in order to meet aggressive test deadlines. One solution to this problem is to provide testers with a test automation framework to facilitate the development of automated test solutions.

Testing machine with workflow based test procedure

Abstract: A test machine system (8) and a method for operating a test machine system (8) includes using a readily available workflow program to a test procedure created using a graphical user interface (47) to arrange test procedure elements

Automation component aspects for efficient unit testing

Abstract: Automation systems software must provide sufficient diagnosis information for testing to enable early defect detection and quality measurement. However, in many automation systems the aspects of automation, testing, and diagnosis are intertwined in the code. This makes the code harder to read, modify, and test. In this paper we introduce the design of a test-driven automation (TDA) component with separate aspects for automation, diagnosis, and testing to improve testability and test efficiency. We illustrate with a prototype, how automation component aspects allow flexible configuration of a “system under test” for test automation. Major result of the pilot application is that the TDA concept was found usable and useful to improve testing efficiency.

Method and apparatus for scheduling a use of test resources of a test arrangement for the execution of test groups

Abstract: In embodiments of the invention methods and apparatus (105) for scheduling the use of test resources of a test arrangement (100) for the execution of test groups are described. The inventive method comprises obtaining (10) an assignment of a test resource of the test arrangement (100) to each test group (TG) of a test flow (5) for testing a device under test (DUT) using the test arrangement (100), wherein the test flow (5) comprises an initial execution order of the test groups. Furthermore, the method comprises checking (20) whether there is a resource conflict (7) between an assignment of a test resource of a test group in a test flow (5) and an assignment of one or more test resources of another test group in another test flow, which other test group is scheduled for a temporally overlapping execution with the given test group using the test arrangement. In addition, the method comprises a step of manipulating (30) the execution order of a test group (TG) in a test flow in response to the result of checking whether there is a resource conflict, such that a resource conflict is eliminated by performing a swap-operation (11) in a test flow between a test group associated with a resource conflict with a higher priority compared to performing a time-interval-insert-operation in combination with a move-operation, moving the test group associated with the resource conflict to an inserted time interval.

System and method for well test design, interpretation and test objectives verification

Abstract: A method and system for well test optimization provides test parameters and variation range therefor, independently from preliminarily reservoir and downhole fluid data; and repeatedly executes the test for each parameter to obtain real-time measured data, and interpreted data that is compared to the variation range for meeting a test objective. A method and system for well test design and interpretation, includes a testing manager having testing hardware/gauge metrology; a geological model; dynamic/static engineering data acquisition; and a reservoir model generator. A method and system for well test design and interpretation generates a test plan and an initial reservoir model and data from real/near-real-time, surface/downhole/manual data, aggregated data based on quality control/assurance, and optimization data based thereon and simulated downhole data; models/interprets the optimization data to meet test objectives for terminating/continuing the test plan; modifies the optimization; and/or generates reports from the modeling/interpretation when terminating the test plan.

Method and system for automated testing of computer applications

Abstract: A system includes a testing device to test a computer application in conjunction with a testing tool, the testing device including a test automation engine to initiate a test, the test automation engine including a programmable data source that is testing tool universal, the programmable data source including one or more automation test scripts that define the test and a data storage device to store results from testing the computer application.

Method and system for testing a software development activity

Abstract: A system and method for testing a software development activity of an enterprise application is disclosed. In one embodiment, the method includes obtaining a software structure specification for an enterprise application and automatically generating test plans using the software structure specification. The method also includes automatically determining at least one test path in each test plan using the software structure specification, and automatically deriving a plurality of test cases using each test path. Further, the method includes executing each of the plurality of test cases and recording test results.

Using continuous integration and automated test techniques for a robust C4ISR system

Abstract: We have used CI (Continuous Integration) and various software testing techniques to achieve a robust C4ISR (Command, Control, Communications, Computers, Intelligence, Surveillance, and Reconnaissance) multi-platform system. Because of rapid changes in the C4ISR domain and in the software technology, frequent critical design adjustments and in turn vast code modifications or additions become inevitable. Defect fixes might also incur code changes. These unavoidable code modifications may put a big risk in the reliability of a mission critical system. Also, in order to stay competitive in the C4ISR market, a company must make recurring releases without sacrificing quality. We have designed and implemented an XML driven automated test framework that enabled us developing numerous high quality tests rapidly. While using CI with automated software test techniques, we have aimed at speeding up the delivery of high quality and robust software by decreasing integration procedure, which is one of the main bottleneck points in the industry. This work describes how we have used CI and software test techniques in a large-scaled, multi-platform, multi-language, distributed C4ISR project and what the benefits of such a system are.

Automatic testing of a computer software system

Abstract: The invention relates to a method of automatic testing of a software system through test driver code that classifies test data into equivalence classes and updates the available test data after using it against the software system. One embodiment of the invention is a Test Runner that monitors the effect of calling the software system on the available test data and uses this information to automatically determine the execution order of test cases to meet a number of objectives including to: Reuse data between calls, ensure all test cases are executed, perform parallelized testing, perform time dependent testing, perform continuous testing according to a probability distribution on test cases, perform automated management of complex test data and finally to provide an easy and concise way for a user to define a large sets of test cases.

Method and system for feature automation

Abstract: A feature automation process defines step-by-step instructions for involving automation engineers, defining, implementing and reviewing software test automation during the development of a feature or product. This process seamlessly integrates the roles of automation engineers and other resources into the software development life cycle (SDLC). An enterprise first creates a dedicated automation team. The feature automation team preferably works with a product/feature team to enable the latter team to better understand the roles of the automation engineers and to further facilitate transparency into the product/feature requirements, design and implementation activities. The feature automation process enables a quality assurance (QA) team to offload (to the feature automation team) the responsibility of writing test scripts, and for creating an automation framework, test designs, and for implementing and maintaining test code. The process ensures that all stakeholders are involved in the reviewing the automation framework and test design prior to test implementation to enhance the reusability of the framework and the stability of the test runs.

A highly configurable test system for evolutionary black-box testing of embedded systems

Abstract: During the development of electronic control units (ECU) in domains like the automotive industry, tests are performed on various test platforms, such as model-in-the-loop, software-in-the-loop, processor-in-the-loop, and hardware-in-the-loop platforms in order to find faults in early development stages. Test cases must be specified to verify the properties demanded of the system on these test platforms. This is an expensive and non-trivial task. Evolutionary black-box testing, a recent approach to automating the creation of interesting test cases, can solve this task completely automatically. This paper describes our evolutionary test system and how to apply it to the test of functional and non-functional properties of embedded systems. Our system supports the aforementioned test platforms and allows for the reuse of the generated test cases across them. In a case study with an antilock braking system, we demonstrate the operation of the system.

FPGA-based low-cost automatic test equipment for digital integrated circuits

Abstract: Digital circuits complexity and density are increasing while, at the same time, more quality and reliability are required. These trends, together with high test costs, make the validation of VLSI circuits more and more difficult. Beside high-end ATE machines, strictly necessary in ASIC production phase, low-cost ATE test systems take place into market to implement a valid support in ASIC development phase. In this paper a case study of low-cost, reconfigurable, versatile and easy-to-use FPGA-based test environment is presented. It allows patterns to be extracted from HDL-simulation and stimuli to be generated to ASIC prototypes, especially when a high-end test machine setup isn't foreseen or isn't available yet. This is the ideal solution for engineers to develop test programs and perform device tests and yield analysis on their desktop and then transfer the test program directly to production. The result is low-cost automatic test equipment, able to execute a preliminary digital test, using just a Laptop and an FPGA-equipped board.

System and method for grammar based test planning

Abstract: The present disclosure generally relates to the testing of a system that includes software or hardware components. In some embodiments, a testing framework generates a set of test cases for a system under test using a grammar. Each test case may perform an action, such as provide an input to the system under test, and result in an output from the system under test. The inputs and outputs are then compared to the expected results to determine whether the system under test is performing correctly. Prior to generating the set of test cases from the grammar, the testing framework processes the grammar to identify attributes of the test cases to be derived from the grammar and facilitates the modification of the grammar.

Test method using memory programmed with tests and protocol to communicate between device under test and tester

Abstract: In an embodiment, a test method is implemented to test an integrated circuit that includes at least one processor The method may include programming a memory to which the integrated circuit is coupled during testing with one or more test programs The integrated circuit may be booted, and the processor may execute the test programs from the memory In one embodiment, the memory may also store a control program that may manage the execution of the tests In an embodiment, the control program may also implement a protocol to communicate with the ATE to perform the testing The protocol may be implemented over a set of general purpose input/output (I/O) pins, for example Using the protocol and test vectors on the ATE, the tests may be selected and executed, and test results may be reported

Reusable Model-Based Testing

Abstract: A reusable model-based testing method for software product lines (SPL) is used to create test specifications from use case and feature models, which can then be configured to test individual applications that are members of the SPL. This paper describes a feature-oriented model-based testing method for SPLs that can be used to reduce the number of reusable test specifications created to cover all use case scenarios, all features, and selected feature combinations of a SPL. These test specifications can be automatically selected and configured during feature-based test derivation to test a given application derived from the SPL. This paper also addresses what application configurations to test and how to configure test specifications for these applications. This model-based testing method was applied and evaluated on two SPL case studies.

Directed random testing

Abstract: Random testing can quickly generate many tests, is easy to implement, scales to large software applications, and reveals software errors. But it tends to generate many tests that are illegal or that exercise the scone parts of the code as other tests, thus limiting its effectiveness. Directed random testing is a new approach to test generation that overcomes these limitations, by combining a bottom-up generation of tests with runtime guidance. A directed random test generator takes a collection of operations under test and generates new tests incrementally, by randomly selecting operations to apply and finding arguments from among previously-constructed tests. As soon as it generates a new test, the generator executes it, and the result determines whether the test is redundant, illegal, error-revealing, or useful for generating more tests. The technique outputs failing tests pointing to potential errors that should be corrected, and passing tests that can be used for regression testing. The thesis also contributes auxiliary techniques that post-process the generated tests, including a simplification technique that transforms a failing test into a smaller one that better isolates the cause of failure, and a branch-directed test generation technique that aims to increase the code coverage achieved by the set of generated tests. Applied to 14 widely-used libraries (including the Java JDK and the core NET framework libraries), directed random testing quickly reveals many serious, previously unknown errors in the libraries. And compared with other test generation tools (model checking, symbolic execution, and traditional random testing), it reveals more errors and achieves higher code coverage. In an industrial case study, a test team at Microsoft using the technique discovered in fifteen hours of human effort as many errors as they typically discover in a person-year of effort using other testing methods. (Copies available exclusively from MIT Libraries, Rm. 14-0551, Cambridge, MA 02139-4307. Ph. 617-253-5668; Fax 617-253-1690.)

Apparatus and method for automatic testing of software or digital devices

Abstract: An apparatus and method for testing a digital device or software installed in the digital device are provided. According to one aspect, the apparatus for testing the digital device or software installed in the digital device includes a test agent for providing a test execution environment, and the test agent performs a test for each test case in response to a command from a test director. The test agent may report an execution state of the test to the test director, and the test director may generate a test result report based on the report or resumes the test upon generation of error.

A Unified Test Framework for Continuous Integration Testing of SOA Solutions

Abstract: The quality of Service Oriented Architecture (SOA) solutions is becoming more and more important along with the increasing adoption of SOA. Continuous Integration Testing (CIT) is an effective technology to discover bugs as early as possible. However, the diversity of programming models used in an SOA solution and the distribution nature of an SOA solution pose new challenges for CIT. Existing testing frameworks more focus on the integration testing of applications developed by a single programming model. In this paper, a unified test framework is proposed to overcome these limitations and enable the CIT of SOA solutions across the whole development lifecycle.This framework is designed following the Model Driven Architecture (MDA). The information of an executable test case is separated into two layers: the behavior layer and the configuration layer. The behavior layer represents the test logic of a test case and is platform independent. The configuration layer contains the platform specific information and is configurable for different programming models. An extensible and pluggable test execution engine is specially designed to execute the integration test cases. A global test case identifier instrumentation approach is used to merge the distributed test case execution traces captured by ITCAM – an IBM integrated management tool. A verification approach supporting Boolean expression and back-end service interaction verification is proposed to verify the test execution result. Initial experiments have shown the effectiveness of this unified test framework.

hol-TestGen

Abstract: We present hol-TestGen , an extensible test environment for specification-based testing build upon the proof assistant Isabelle. hol-TestGen leverages the semi-automated generation of test theorems (a form of partitioning the test input space), and their refinement to concrete test-data, as well as the automatic generation of a test driver for the execution and test result verification. hol-TestGen can also be understood as a unifying technical and conceptual framework for presenting and investigating the variety of unit test and sequence test techniques in a logically consistent way.

Synthesizing Test Models from Test Cases

Abstract: In this paper we describe a methodology for synthesizing test models from test cases. The context of our approach is model-based graphical user interface (GUI) testing of smartphone applications. To facilitate the deployment of model-based testing practices, existing assets in test automation should be utilized. While companies are interested in the benefits of new approaches, they may have already invested heavily in conventional test suites. The approach presented in this paper enables using such suites for creating complex test models that should have better defect detection capability. The synthesis is illustrated with examples from two small case studies conducted using real test cases from industry. Our approach is semi-automatic requiring user interaction. We also outline planned tool support to enable efficient synthesis process.