Study of ERP Test-Suite Reduction Based on Modified Condition/Decision Coverage
07 May 2010-pp 373-378
TL;DR: This paper proposes a new test-suite reduction technique for MC/DC: a bi-objective model that considers both the coverage degree of test case for test requirements and the capability of test cases to reveal error, and results show that the technique both reduces the size of test suite and better ensures the effectiveness of test suites to revealerror.
Abstract: Enterprise Resource Planning (ERP) systems represent a huge market in the commercial arena. Products from suppliers such as SAP, Oracle and more recently, Microsoft, dominate the software market. Testing in these projects is a significant effort but is hardly supported by methods and tools other than those provided by the suppliers themselves. Experience shows that testing in these projects is critical, but often neglected. Recent 'lessons learned' work by the Paul Gerrard indicates that a benefit, risk- and coverage-based test approach could significantly reduce the risk of failures. It is evidence that modified condition/decision coverage (MC/DC) is an effective verification method and can help to detect safety faults despite of its expensive cost. In regression testing, it is quite costly to return all of test cases in test suite because new test cases are added to test suite as the software evolves. Therefore, it is necessary to reduce the test suite to improve test efficiency and save test cost. Many existing test-suite reduction techniques are not effective to reduce MC/DC test suite. This paper proposes a new test-suite reduction technique for MC/DC: a bi-objective model that considers both the coverage degree of test case for test requirements and the capability of test cases to reveal error. Our experiment results show that the technique both reduces the size of test suite and better ensures the effectiveness of test suite to reveal error.
Citations
More filters
[...]
TL;DR: An overview of existing approaches that are able to reduce testing effort is presented both for researchers and practitioners in order to identify, in the one hand, future research directions and, on the other hand, potential for improvements in practical environments.
Abstract: Context: Quality assurance effort, especially testing effort, is often a major cost factor during software development, which sometimes consumes more than 50% of the overall development effort. Consequently, one major goal is often to reduce testing effort. Objective: The main goal of the systematic mapping study is the identification of existing approaches that are able to reduce testing effort. Therefore, an overview should be presented both for researchers and practitioners in order to identify, on the one hand, future research directions and, on the other hand, potential for improvements in practical environments. Method: Two researchers performed a systematic mapping study, focusing on four databases with an initial result set of 4020 articles. Results: In total, we selected and categorized 144 articles. Five different areas were identified that exploit different ways to reduce testing effort: approaches that predict defect-prone parts or defect content, automation, test input reduction approaches, quality assurance techniques applied before testing, and test strategy approaches. Conclusion: The results reflect an increased interest in this topic in recent years. A lot of different approaches have been developed, refined, and evaluated in different environments. The highest attention was found with respect to automation and prediction approaches. In addition, some input reduction approaches were found. However, in terms of combining early quality assurance activities with testing to reduce test effort, only a small number of approaches were found. Due to the continuous challenge of reducing test effort, future research in this area is expected.
66 citations
Cites methods from "Study of ERP Test-Suite Reduction B..."
[...]
[...]
TL;DR: The findings suggest that hybrid criteria of others can be described using the Merge and Rank formulations, and that the hybrid criteria the authors developed most often outperformed their constituent individual criteria.
Abstract: Regression testing tasks of test case prioritization, test suite reduction/minimization, and regression test selection are typically centered around criteria that are based on code coverage, test execution costs, and code modifications. Researchers have developed and evaluated new individual criteria; others have combined existing criteria in different ways to form what we--and some others--call hybrid criteria. In this paper, we formalize the notion of combining multiple criteria into a hybrid. Our goal is to create a uniform representation of such combinations so that they can be described unambiguously and shared among researchers. We envision that such sharing will allow researchers to implement, study, extend, and evaluate the hybrids using a common set of techniques and tools. We precisely formulate three hybrid combinations, Rank, Merge, and Choice, and demonstrate their usefulness in two ways. First, we recast, in terms of our formulations, others' previously reported work on hybrid criteria. Second, we use our previous results on test case prioritization to create and evaluate new hybrid criteria. Our findings suggest that hybrid criteria of others can be described using our Merge and Rank formulations, and that the hybrid criteria we developed most often outperformed their constituent individual criteria.
48 citations
Cites background or methods from "Study of ERP Test-Suite Reduction B..."
[...]
[...]
[...]
TL;DR: A specification driven approach to test GUI-Based java programs as an alternative to the CP technique is proposed and a GUI-event test specification language based on Java Script is introduced from which an automated test engine is generated.
Abstract: A pervasive method for GUI testing is the Capture and Playback (CP) technique. This commonly used technique cannot be used until an Application Under Test (AUT) is completely developed. In this paper we propose a specification driven approach to test GUI-Based java programs as an alternative to the CP technique. We introduce a GUI-event test specification language based on Java Script from which an automated test engine is generated. The esteem of Java as a scripting language is its ease of use and its standard format that have made writing a test script using our proposed specification language makes it an easy task. Beside the ability to test AUT before being completely developed we have implemented our approach that can generate the test specification file for an already existing AUT. The Tool generates GUI events, where Captures and Playback event responses to automatic verification of the results for the test cases which are written to a test log file. This approach supports M-version testing, where each version of the application is intended to satisfy the same specification.
4 citations
Cites background from "Study of ERP Test-Suite Reduction B..."
[...]
[...]
TL;DR: This research paper proposes the new model that is used for improving the regression test reduction, which guarantees the best results compared with traditional methods by adding four algorithms; testing, ordering, deleting test cases and fixing bugs.
Abstract: Problem of test suite size increases in the process of software maintenance when numbers of revision are required can be solved by reduction algorithms such random and regression test reduction. Nevertheless, the whole performance need to be improved. Therefore, this research paper proposes the new model that is used for improving the regression test reduction, which guarantees the best results compared with traditional methods by adding four algorithms; testing, ordering, deleting test cases and fixing bugs. Accordingly, it gives smaller size and higher percent fixing bugs as about 40% and 200% respectively.
Cites background from "Study of ERP Test-Suite Reduction B..."
[...]
References
More filters
[...]
TL;DR: A technique to select a representative set of test cases from a test suite that provides the same coverage as the entire test suite by identifying, and then eliminating, the redundant and obsolete test cases in the test suite is presented.
Abstract: This paper presents a technique to select a representative set of test cases from a test suite that provides the same coverage as the entire test suite. This selection is performed by identifying, and then eliminating, the redundant and obsolete test cases in the test suite. The representative set replaces the original test suite and thus, potentially produces a smaller test suite. The representative set can also be used to identify those test cases that should be rerun to test the program after it has been changed. Our technique is independent of the testing methodology and only requires an association between a testing requirement and the test cases that satisfy the requirement. We illustrate the technique using the data flow testing methodology. The reduction that is possible with our technique is illustrated by experimental results.
630 citations
[...]
TL;DR: The authors present a technique that selects from a test suite a representative set of test cases that provides the same measure of coverage as the test suite by means of the data flow testing methodology.
Abstract: As a result of modifications to a program during the maintenance phase, the size of a test suite used for regression testing can become unmanageable. The authors present a technique that selects from a test suite a representative set of test cases that provides the same measure of coverage as the test suite. This selection is performed by the identification of the redundant and obsolete test cases in the test suite. The representative set can be used to reduce the size of the test suite by substituting for the test suite. The representative set can also be used to determine those test cases that should be rerun to test the program after it has been changed. The technique is independent of the testing methodology and only requires an association between each testing requirement and the test cases that satisfy the requirement. The technique is illustrated by means of the data flow testing methodology. Experimental studies are being performed that demonstrate the effectiveness of the technique. >
497 citations
"Study of ERP Test-Suite Reduction B..." refers methods in this paper
[...]
[...]
[...]
TL;DR: New algorithms for test-suite reduction and prioritization that can be tailored effectively for use with modified condition/decision coverage (MC/DC) adequate are presented.
Abstract: Software testing is particularly expensive for developers of high-assurance software, such as software that is produced for commercial airborne systems. One reason for this expense is the Federal Aviation Administration's requirement that test suites be modified condition/decision coverage (MC/DC) adequate. Despite its cost, there is evidence that MC/DC is an effective verification technique and can help to uncover safety faults. As the software is modified and new test cases are added to the test suite, the test suite grows and the cost of regression testing increases. To address the test-suite size problem, researchers have investigated the use of test-suite reduction algorithms, which identify a reduced test suite that provides the same coverage of the software according to some criterion as the original test suite, and test-suite prioritization algorithms, which identify an ordering of the test cases in the test suite according to some criteria or goals. Existing test-suite reduction and prioritization techniques, however, may not be effective in reducing or prioritizing MC/DC-adequate test suites because they do not consider the complexity of the criterion. This paper presents new algorithms for test-suite reduction and prioritization that can be tailored effectively for use with MC/DC. The paper also presents the results of empirical studies of these algorithms.
373 citations
[...]
TL;DR: This study presents a new metric for assessing the rate of fault detection of prioritized test cases that incorporates varying test case and fault costs and presents the results of a case study illustrating the application of the metric.
Abstract: Test case prioritization techniques schedule test cases for regression testing in an order that increases their ability to meet some performance goal. One performance goal, rate of fault detection, measures how quickly faults are detected within the testing process. In previous work (S. Elbaum et al., 2000; G. Rothermel et al., 1999), we provided a metric, APFD, for measuring rate of fault detection, and techniques for prioritizing test cases to improve APFD, and reported the results of experiments using those techniques. This metric and these techniques, however, applied only in cases in which test costs and fault severity are uniform. We present a new metric for assessing the rate of fault detection of prioritized test cases that incorporates varying test case and fault costs. We present the results of a case study illustrating the application of the metric. This study raises several practical questions that might arise in applying test case prioritization; we discuss how practitioners could go about answering these questions.
349 citations
[...]
TL;DR: An experiment in which the costs and benefits of minimizing test suites of various sizes for several programs are compared reveals that the fault detection capabilities of test suites can be severely compromised by minimization.
Abstract: Test suite minimization techniques attempt to reduce the cost of saving and reusing tests during software maintenance, by eliminating redundant tests from test suites. A potential drawback of these techniques is that in minimizing a test suite, they might reduce the ability of that test suite to reveal faults in the software. A study showed that minimization can reduce test suite size without significantly reducing the fault detection capabilities of test suites. To further investigate this issue, we performed an experiment in which we compared the costs and benefits of minimizing test suites of various sizes for several programs. In contrast to the previous study, our results reveal that the fault detection capabilities of test suites can be severely compromised by minimization.
304 citations
"Study of ERP Test-Suite Reduction B..." refers background or methods in this paper
[...]
[...]
[...]
Related Papers (5)
[...]
[...]