Showing papers on "White-box testing published in 2010"

Software testing

Abstract: Software testing is any activity aimed at evaluating an attribute or capability of a program or system and determining that it meets its required results. Although crucial to software quality and widely deployed by programmers and testers, software testing still remains an art, due to limited understanding of the principles of software. The difficulty in software testing stems from the complexity of software: we can not completely test a program with moderate complexity. Testing is more than just debugging. The purpose of testing can be quality assurance, verification and validation, or reliability estimation. Testing can be used as a generic metric as well. Correctness testing and reliability testing are two major areas of testing. Software testing is a trade-off between budget, time and quality.

Cloud9: a software testing service

Abstract: Cloud9 aims to reduce the resource-intensive and laborintensive nature of high-quality software testing. First, Cloud9 parallelizes symbolic execution (an effective, but still poorly scalable test automation technique) to large shared-nothing clusters. To our knowledge, Cloud9 is the first symbolic execution engine that scales to large clusters of machines, thus enabling thorough automated testing of real software in conveniently short amounts of time. Preliminary results indicate one to two orders of magnitude speedup over a state-of-the-art symbolic execution engine. Second, Cloud9 is an on-demand software testing service: it runs on compute clouds, like Amazon EC2, and scales its use of resources over a wide dynamic range, proportionally with the testing task at hand.

Automated incremental pairwise testing of software product lines

Abstract: Testing Software Product Lines is very challenging due to a high degree of variability leading to an enormous number of possible products. The vast majority of today's testing approaches for Software Product Lines validate products individually using different kinds of reuse techniques for testing. Due to the enormous number of possible products, individual product testing becomes more and more unfeasible. Combinatorial testing offers one possibility to test a subset of all possible products. In this contribution we provide a detailed description of a methodology to apply combinatorial testing to a feature model of a Software Product Line. We combine graph transformation, combinatorial testing, and forward checking for that purpose. Additionally, our approach considers predefined sets of products.

An Automatic Testing Approach for Compiler Based on Metamorphic Testing Technique

Abstract: Compilers play an important role in software development, and it is quite necessary to perform abundant testing to ensure the correctness of compilers. A critical task in compiler testing is to validate the semantic-soundness property which requires consistence between semantics of source programs and behavior of target executables. For validating this property, one main challenging issue is generation of a test oracle. Most existing approaches fall into two main categories when dealing with this issue: reference-based approaches and assertion-based approaches. All these approaches have their weakness when new programming languages are involved or test automation is required. To overcome the weakness in the existing approaches, we propose a new automatic approach for testing compiler. Our approach is based on the technique of metamorphic testing, which validates software systems via so-called "metamorphic relations". We select the equivalence-preservation relation as the metamorphic relation and propose an automatic metamorphic testing framework for compiler. We also propose three different techniques for automatically generating equivalent source programs as test inputs. Based on our approach, we developed a tool called Mettoc. Our mutation experiments show that Mettoc is effective to reveal compilers' errors in terms of the semantic-soundness property. Moreover, the empirical results also reveal that simple approaches for constructing test inputs are not weaker than complicated ones in terms of fault-detection capability. We also applied Mettoc in testing a number of open source compilers, and two real errors in GCC-4.4.3 and UCC-1.6 respectively have been detected by Mettoc.

Formal analysis of the effectiveness and predictability of random testing

Abstract: There has been a lot of work to shed light on whether random testing is actually a useful testing technique. Despite its simplicity, several successful real-world applications appear in the literature. Although it is not going to solve all possible testing problems, random testing is an essential tool in the hands of software testers. In this paper, we address general questions about random testing, such as how long random testing needs on average to achieve testing targets (e.g., coverage), how does it scale and how likely is it to yield similar results if we re-run random testing on the same testing problem. Due to its simplicity that makes the mathematical analysis of random testing tractable, we provide precise and rigorous answers to these questions. Our formal results can be applied to most types of software and testing criteria. Simulations are carried out to provide further support to our formal results. The obtained results are then used to assess the validity of empirical analyses reported in the literature. Results show that random testing is more effective and predictable than previously thought.

Efficient Software Verification: Statistical Testing Using Automated Search

Abstract: Statistical testing has been shown to be more efficient at detecting faults in software than other methods of dynamic testing such as random and structural testing. Test data are generated by sampling from a probability distribution chosen so that each element of the software's structure is exercised with a high probability. However, deriving a suitable distribution is difficult for all but the simplest of programs. This paper demonstrates that automated search is a practical method of finding near-optimal probability distributions for real-world programs, and that test sets generated from these distributions continue to show superior efficiency in detecting faults in the software.

Black-box system testing of real-time embedded systems using random and search-based testing

Abstract: Testing real-time embedded systems (RTES) is in many ways challenging. Thousands of test cases can be potentially executed on an industrial RTES. Given the magnitude of testing at the system level, only a fully automated approach can really scale up to test industrial RTES. In this paper we take a black-box approach and model the RTES environment using the UML/- MARTE international standard. Our main motivation is to provide a more practical approach to the model-based testing of RTES by allowing system testers, who are often not familiar with the system design but know the application domain well-enough, to model the environment to enable test automation. Environment models can support the automation of three tasks: the code generation of an environment simulator, the selection of test cases, and the evaluation of their expected results (oracles). In this paper, we focus on the second task (test case selection) and investigate three test automation strategies using inputs from UML/MARTE environment models: Random Testing (baseline), Adaptive Random Testing, and Search-Based Testing (using Genetic Algorithms). Based on one industrial case study and three artificial systems, we show how, in general, no technique is better than the others. Which test selection technique to use is determined by the failure rate (testing stage) and the execution time of test cases. Finally, we propose a practical process to combine the use of all three test strategies.

Large-Scale Software Testing Environment Using Cloud Computing Technology for Dependable Parallel and Distributed Systems

Abstract: Various information systems are widely used in information society era, and the demand for highly dependable system is increasing year after year. However, software testing for such a system becomes more difficult due to the enlargement and the complexity of the system. In particular, it is too difficult to test parallel and distributed systems sufficiently although dependable systems such as high-availability servers usually form parallel and distributed systems. To solve these problems, we proposed a software testing environment for dependable parallel and distributed system using the cloud computing technology, named D-Cloud. D-Cloud includes Eucalyptus as the cloud management software, and FaultVM based on QEMU as the virtualization software, and D-Cloud frontend for interpreting test scenario. D-Cloud enables not only to automate the system configuration and the test procedure but also to perform a number of test cases simultaneously, and to emulate hardware faults flexibly. In this paper, we present the concept and design of D-Cloud, and describe how to specify the system configuration and the test scenario. Furthermore, the preliminary test example as the software testing using D-Cloud was presented. Its result shows that D-Cloud allows to set up the environment easily, and to test the software testing for the distributed system.

Automated analysis of load testing results

Abstract: Many software systems must be load tested to ensure that they can scale up while maintaining functional and performance requirements. Current industrial practices for checking the results of a load test remain ad hoc, involving high level checks. Few research efforts are devoted to the automated analysis of load testing results, mainly due to the limited access to large scale systems for use as case studies. Automated and systematic load testing analysis is going to be much needed, as many services have been offered online to an increasing number of users. This dissertation proposes automated approaches to detect functional and performance problems in a load test by mining the recorded load testing data (execution logs and performance metrics). Case studies show that our approaches scale well to large enterprise systems and output high precision results that help analysts detect load testing problems.

Recovering traceability links between unit tests and classes under test: An improved method

Abstract: Unit tests are valuable as a source of up-to-date documentation as developers continuously changes them to reflect changes in the production code to keep an effective regression suite. Maintaining traceability links between unit tests and classes under test can help developers to comprehend parts of a system. In particular, unit tests show how parts of a system are executed and as such how they are supposed to be used. Moreover, the dependencies between unit tests and classes can be exploited to maintain the consistency during refactoring. Generally, such dependences are not explicitly maintained and they have to be recovered during software development. Some guidelines and naming conventions have been defined to describe the testing environment in order to easily identify related tests for a programming task. However, very often these guidelines are not followed making the identification of links between unit tests and classes a time-consuming task. Thus, automatic approaches to recover such links are needed. In this paper a traceability recovery approach based on Data Flow Analysis (DFA) is presented. In particular, the approach retrieves as tested classes all the classes that affect the result of the last assert statement in each method of the unit test class. The accuracy of the proposed method has been empirically evaluated on two systems, an open source system and an industrial system. As a benchmark, we compare the accuracy of the DFA-based approach with the accuracy of the previously used traceability recovery approaches, namely Naming Convention (NC) and Last Call Before Assert (LCBA) that seem to provide the most accurate results. The results show that the proposed approach is the most accurate method demonstrating the effectiveness of DFA. However, the case study also highlights the limitations of the experimented traceability recovery approaches, showing that detecting the class under test cannot be fully automated and some issues are still under study.

Increasing functional coverage by inductive testing: a case study

Abstract: This paper addresses the challenge of generating test sets that achieve functional coverage, in the absence of a complete specification. The inductive testing technique works by probing the system behaviour with tests, and using the test results to construct an internal model of software behaviour, which is then used to generate further tests. The idea in itself is not new, but prior attempts to implement this idea have been hampered by expense and scalability, and inflexibility with respect to testing strategies. In the past, inductive testing techniques have tended to focus on the inferred models, as opposed to the suitability of the test sets that were generated in the process. This paper presents a flexible implementation of the inductive testing technique, and demonstrates its application with case-study that applies it to the Linux TCP stack implementation. The evaluation shows that the generated test sets achieve a much better coverage of the system than would be achieved by similar non-inductive techniques.

A Model-Based Regression Testing Approach for Evolving Software Systems with Flexible Tool Support

Abstract: Model-based selective regression testing promises reduction in cost and labour by selecting a subset of the test suite corresponding to the modifications after system evolution. However, identification of modifications in the systems and selection of corresponding test cases is challenging due to interdependencies among models. State-based testing is an important approach to test the system behaviour. Unfortunately the existing state-based regression testing approaches do not care for dependencies of the state machines with other system models. This paper presents the tool support and evaluation of our state-based selective regression testing methodology for evolving state-based systems. START is an Eclipse-based tool for state-based regression testing compliant with UML 2.1 semantics. START deals with dependencies of state machines with class diagrams to cater for the change propagation. We applied the START on a case study and our results show significant reduction in the test cases resulting in reduction in testing time and cost.

Statistical Path Selection for At-Speed Test

Abstract: Process variations make at-speed testing significantly more difficult. They cause subtle delay changes that are distributed in contrast to the localized nature of a traditional fault model. Due to parametric variations, different paths can be critical in different parts of the process space, and the union of such paths must be tested to obtain good process space coverage. This paper proposes an integrated at-speed structural testing methodology, and develops a novel branch-and-bound algorithm that elegantly and efficiently solves the hitherto open problem of statistical path tracing. The resulting paths are used for at-speed structural testing. A new test quality metric is proposed, and paths which maximize this metric are selected. After chip timing has been performed, the path selection procedure is extremely efficient. Path selection for a multimillion gate chip design can be completed in a matter of seconds.

Longer is Better: On the Role of Test Sequence Length in Software Testing

Abstract: In the presence of an internal state, often it is required a sequence of function calls to test software. In fact, to cover a particular branch of the code, a sequence of previous function calls might be required to put the internal state in the appropriate configuration. Internal states are not only present in object-oriented software, but also in procedural software(e.g., static variables in C programs). In the literature, there are many techniques to test this type of software. However, to our best knowledge, the properties related to choosing the length of these sequences have received only little attention in the literature. In this paper, we analyse the role that the length plays in software testing, in particular branch coverage. We show that on “difficult” software testing benchmarks longer test sequences make their testing trivial. Hence, we argue that the choice of the length of the test sequences is very important in software testing.

Towards automating the generation of mutation tests

Abstract: Automating software testing activities can increase the quality and drastically decrease the cost of software development. Towards this direction various automated test data generation tools have been developed. The majority of them aim at branch testing, while a quite limited number aim at a higher level of testing thoroughness such as mutation. In this paper an automated framework that makes a joint use of diverse techniques and tools is introduced in the context of automating mutation based test generation. The motivation behind this work is the use of existing techniques and tools such as symbolic execution and evolutionary testing towards automating the test input generation activity according to the weak mutation testing criterion. The proposed framework integrates existing automated tools for branch testing in order to effectively generate mutation test data. To fulfill this suggestion three automated tools are used for illustration purposes and preliminary results are obtained by applying the proposed framework to a set of java program units indicating the applicability and effectiveness of the proposed approach.

Simulation-based interface testing automation system and method for robot software components

Abstract: A testing tool for verifying a reliability of robot software components is provided, and more particularly, a simulation-based interface testing automation system and method that may automatically perform a test for functions and performance of an interface with respect to robot software components are provided. Thus, it is possible to efficiently perform a test for an interface of robot software components. In particular, it is possible to automatically generate test cases required for testing, to perform a test in various languages, and to perform a test suitable for a robot.

Automatically redirecting method calls for unit testing

Abstract: A unit testing system allows testing of multiple types of method calls using an unmodified software application module. The system dynamically copies byte code of the tested method, clones the byte code, and executes the cloned byte code. During cloning, the system instruments the code so that method calls are redirected into a system method, where the system can determine whether a mocked method should be called instead. The result is that the developer does not need to modify the original methods in order to redirect method calls to mock methods. The developer can simply reference a test module of the unit testing system and easily write tests that redirect any method they wish. Thus, the unit testing system provides a framework for building more useful unit tests that cause less interference with production-ready code.

Reducing the cost of model-based testing through test case diversity

Abstract: Model-based testing (MBT) suffers from two main problems which in many real world systems make MBT impractical: scalability and automatic oracle generation. When no automated oracle is available, or when testing must be performed on actual hardware or a restricted-access network, for example, only a small set of test cases can be executed and evaluated. However, MBT techniques usually generate large sets of test cases when applied to real systems, regardless of the coverage criteria. Therefore, one needs to select a small enough subset of these test cases that have the highest possible fault revealing power. In this paper, we investigate and compare various techniques for rewarding diversity in the selected test cases as a way to increase the likelihood of fault detection. We use a similarity measure defined on the representation of the test cases and use it in several algorithms that aim at maximizing the diversity of test cases. Using an industrial system with actual faults, we found that rewarding diversity leads to higher fault detection compared to the techniques commonly reported in the literature: coverage-based and random selection. Among the investigated algorithms, diversification using Genetic Algorithms is the most cost-effective technique.

Goal-oriented testing for MASs

Abstract: As Multi-Agent Systems (MASs) are increasingly applied in complex distributed applications such as financial and healthcare services, assurance needs to be given to the user that the implemented MASs operate properly, i.e., they meet their specifications and the stakeholder's expectations. Testing is an important and widely applied technique in practice to reach this goal. Current Agent-Oriented Software Engineering (AOSE) methodologies address testing only partially. Some of them exploit Object-Oriented (OO) testing techniques, based on a mapping of agent-oriented abstractions into OO constructs; this may require additional development effort and introduce anomalies. Moreover, a structured testing process for AOSE methodologies is still missing. In this paper we introduce a testing methodology, called Goal-Oriented Software Testing (GOST), for MAS. The methodology specifies a testing process that complements goal-oriented analysis and design. Furthermore, GOST provides a systematic way of deriving test cases from goal-oriented specifications and techniques to automate test case generation and their execution.

A platform for search-based testing of concurrent software

Abstract: The paper describes a generic, open-source infrastructure called SearchBestie (or S'Bestie for short) that we propose as a platform for experimenting with search-based techniques and for applying them in the area of software testing. Further, motivated by a lack of work on search-based testing targeted at identifying concurrency-related problems, we instantiate S'Bestie for search-based testing of concurrent programs using the IBM's concurrency testing infrastructure called ConTest. We demonstrate on series of experiments the capabilities of S'Bestie and (despite we have just made our first experiments with S'Bestie) also the fact that using search-based testing in the context of testing concurrent programs can be useful.

A method for analyzing system state-space coverage within a t-wise testing framework

Abstract: Inadequate state-space coverage of complex configurable systems during test phases is an area of concern for systems engineers. Determining the state-space coverage of a proposed or executed test suite traditionally involves qualitative assessment, rendering meaningful comparative analysis between tests for a given system or across multiple systems difficult. We propose a method for assessing state-space coverage of a test suite utilizing t-wise testing, a combinatorial technique borrowed from the software testing community which generalizes pair-wise testing. We refine traditional notions of a t-wise test suite to analyze the configuration coverage of a test plan. This provides a methodology and a set of metrics to assess both the level and the distribution of state-space coverage. We detail a proof-of-concept experiment using this partial t-wise coverage framework to analyze Integration and Test (I&T) data from three separate NASA spacecraft.

Online Testing Framework for Web Services

Abstract: Testing conceptually consists of three activities: test case generation, test case execution and verdict assignment. Using online testing, test cases are generated and simultaneously executed (i.e. the complete test scenario is built during test execution). This paper presents a framework that automatically generates and executes tests "online" for conformance testing of a composite of Web services described in BPEL. The proposed framework considers unit testing and it is based on a timed modeling of BPEL specification, a distributed testing architecture and an online testing algorithm that generates, executes and assigns verdicts to every generated state in the test case.

Avoiding redundant testing in application engineering

Abstract: Many software product line testing techniques have been presented in the literature The majority of those techniques address how to define reusable test assets (such as test models or test scenarios) in domain engineering and how to exploit those assets during application engineering In addition to test case reuse however, the execution of test cases constitutes one important activity during application testing Without a systematic support for the test execution in application engineering, while considering the specifics of product lines, product line artifacts might be tested redundantly Redundant testing in application engineering, however, can lead to an increased testing effort without increasing the chance of uncovering failures In this paper, we propose the model-based ScenTED-DF technique to avoid redundant testing in application engineering Our technique builds on data flow-based testing techniques for single systems and adapts and extends those techniques to consider product line variability The paper sketches the prototypical implementation of our technique to show its general feasibility and automation potential, and it describes the results of experiments using an academic product line to demonstrate that ScenTED-DF is capable of avoiding redundant tests in application engineering

A Systematic Capture and Replay Strategy for Testing Complex GUI Based Java Applications

Abstract: In this paper we propose a testing strategy that targets Java applications with complex GUI structure and event interactions. We present a capture and replay testing technique which can be employed for different testing purposes: GUI convergence, functional testing and regression testing. The proposed strategy drastically improves, and from different aspects, on standard capture and replay tools. This is done by combining both a model based testing approach with the capture and replay approach and by implementing different automated test oracles. We first model the behavior of the system from the functional specifications or from a trusted version of the system. Tests are then derived from this model to exercise the system in order to ensure correct functional behavior and to cover goal oriented interactions. The case study applies the test strategy on a role-based, multi-user computer game to demonstrate the usefulness and importance of this approach.

Using Bounded Model Checking for Coverage Analysis of Safety-Critical Software in an Industrial Setting

Abstract: Testing and Bounded Model Checking (BMC) are two techniques used in Software Verification for bug-hunting. They are expression of two different philosophies: testing is used on the compiled code and it is more suited to find errors in common behaviors, while BMC is used on the source code to find errors in uncommon behaviors of the system. Nowadays, testing is by far the most used technique for software verification in industry: it is easy to use and even when no error is found, it can release a set of tests certifying the (partial) correctness of the compiled system. In the case of safety critical software, in order to increase the confidence of the correctness of the compiled system, it is often required that the provided set of tests covers 100% of the code. This requirement, however, substantially increases the costs associated to the testing phase, since it often involves the manual generation of tests. In this paper we show how BMC can be productively applied to the Software Verification process in industry. In particular, we show how to productively use a Bounded Model Checker for C programs (CBMC) as an automatic test generator for the Coverage Analysis of Safety Critical Software. In particular, we experimented CBMC on a subset of the modules of the European Train Control System (ETCS) of the European Rail Traffic Management System (ERTMS) source code, an industrial system for the control of the traffic railway, provided by Ansaldo STS. The Code of the ERTMS/ETCS, with thousands of lines, has been used as trial application with CBMC obtaining a set of tests satisfying the target 100% code coverage, requested by the CENELEC EN50128 guidelines for software development of safety critical systems. The use of CBMC for test generation led to a dramatic increase in the productivity of the entire Software Development process by substantially reducing the costs of the testing phase. To the best of our knowledge, this is the first time that BMC techniques have been used in an industrial setting for automatically generating tests achieving full coverage of Safety-Critical Software. The positive results demonstrate the maturity of Bounded Model Checking techniques for automatic test generation in industry.

Model-based testing in industry: a case study with two MBT tools

Abstract: Traditional testing techniques may not always be suitable for adequate, thorough, and extensible testing of critical and complex software in a resource and time constrained software development environment. Model-based testing (MBT) is an evolving technique for generating test cases automatically from a behavioral model of a system under test. For a Siemens industrial project in the healthcare domain high quality, reliable and robust software is indispensable. Thus, we must ensure rigorous testing using a model based approach. We specify nine essential criteria to be satisfied to ensure the quality of the medical software with MBT. We have selected two recently launched MBT tools, namely Conformiq's Qtronic 2.0 and Microsoft's SpecExplorer 2010 V3.0, to see how far the criteria set for testing are achieved by these two tools. This paper reports on the evaluation of the tools and discusses the scope and objectives of future generation MBT tools.

Test automation on mobile device

Abstract: To ensure the software quality of mobile device, it is essential to build sophisticated testing automation strategy. In this paper, we introduced the idea of constructing a testing framework, elaborated the problems encountered in practice and proposed the solutions accordingly, where several innovative testing methods and tactics were brought forward and discussed in details. Furthermore, the testing framework was used for testing a tailor-made program on mobile phone. Testing result analysis proves that the framework is not only viable and effective on mobile device testing, but also shows that the testing efficiency needs to be improved.

Generating Feasible Test Paths from an Executable Model Using a Multi-objective Approach

Abstract: Search-based testing techniques using meta-heuristics, like evolutionary algorithms, has been largely used for test data generation, but most approaches were proposed for white-box testing. In this paper we present an evolutionary approach for test sequence generation from a behavior model, in particular, Extended Finite State Machine. An open problem is the production of infeasible paths, as these should be detected and discarded manually. To circumvent this problem, we use an executable model to obtain feasible paths dynamically. An evolutionary algorithm is used to search for solutions that cover a given test purpose, which is a transition of interest. The target transition is used as a criterion to get slicing information, in this way, helping to identify the parts of the model that affect the test purpose. We also present a multi-objective search: the test purpose coverage and the sequence size minimization, as longer sequences require more effort to be executed.