Proceedings ArticleDOI
Software performance testing using covering arrays: efficient screening designs with categorical factors
Dean S. Hoskins,Charles J. Colbourn,Douglas C. Montgomery +2 more
- pp 131-136
TLDR
Commercial covering array generators, while not as good as exhaustively generated designs, remain competitive with D-optimal design generators to compare designs.Abstract:
Classical Design of Experiment (DOE) techniques have been in use for many years to aid in the performance testing of systems. In particular fractional factorial designs have been used in cases with many numerical factors to reduce the number of experimental runs necessary. For experiments involving categorical factors, this is not the case; experimenters regularly resort to exhaustive (full factorial) experiments. Recently, D-optimal designs have been used to reduce numbers of tests for experiments involving categorical factors because of their flexibility, but not necessarily because they can closely approximate full factorial results. In commonly used statistical packages, the only generic alternative for reduced experiments involving categorical factors is afforded by optimal designs. The extent to which D-optimal designs succeed in estimating exhaustive results has not been evaluated, and it is natural to determine this. An alternative design based on covering arrays may offer a better approximation of full factorial data. Covering arrays are used in software testing for accurate coverage of interactions, while D-optimal and factorial designs measure the amount of interaction. Initial work involved exhaustive generation of designs in order to compare covering arrays and D-optimal designs in approximating full factorial designs. In that setting, covering arrays provided better approximations of full factorial analysis, while ensuring coverage of all small interactions. Here we examine commercially viable covering array and D-optimal design generators to compare designs. Commercial covering array generators, while not as good as exhaustively generated designs, remain competitive with D-optimal design generators.read more
Citations
More filters
Journal ArticleDOI
Prioritized interaction testing for pair-wise coverage with seeding and constraints☆
Renee Bryce,Charles J. Colbourn +1 more
TL;DR: A ‘‘one-test-at-a-time’’ greedy method is adapted to take importance of pairs into account, so that when run to completion all pair-wise interactions are tested, but when terminated after any intermediate number of tests, those deemed most important are tested.
Journal ArticleDOI
A Survey on Load Testing of Large-Scale Software Systems
Zhen Ming Jiang,Ahmed E. Hassan +1 more
TL;DR: The state of load testing research and practice is surveyed and current techniques that are used in the three phases of a load test are compared and contrast.
Journal ArticleDOI
Locating and detecting arrays for interaction faults
TL;DR: In this article, the problem of nonadaptive location of interaction faults is formalized under the hypothesis that the system contains (at most) some number d of faults, each involving ( at most) a number t of interacting factors.
Journal ArticleDOI
A variable strength interaction test suites generation strategy using Particle Swarm Optimization
Bestoun S. Ahmed,Kamal Z. Zamli +1 more
TL;DR: Comparative results indicate that VS-PSTG gives competitive results as compared to existing strategies, and adopts Particle Swarm Optimization to ensure optimal test size reduction.
Journal ArticleDOI
Application of Particle Swarm Optimization to uniform and variable strength covering array construction
TL;DR: The effectiveness of the proposed Particle Swarm-based t-way Test Generator (PSTG) for generating uniform and variable strength covering arrays and the usefulness of PSTG for facilitating fault detection owing to interactions of the input components is demonstrated.
References
More filters
Journal ArticleDOI
The AETG system: an approach to testing based on combinatorial design
TL;DR: A new approach to testing that uses combinatorial designs to generate tests that cover the pairwise, triple, or n-way combinations of a system's test parameters, and is implemented in the AETG system.
Journal ArticleDOI
An algorithm for the construction of “ D -optimal” experimental designs
TL;DR: The algorithm DETMAX is presented, whose purpose is to construct experimental designs that are “D-optimal,” which are designs for which the determinant of X'X is maximum.
Journal ArticleDOI
The Coordinate-Exchange Algorithm for Constructing Exact Optimal Experimental Designs
TL;DR: The algorithm uses a variant of the Gauss-Southwell cyclic coordinate-descent algorithm within the k-exchange algorithm to achieve substantive reductions in required computing.
Journal ArticleDOI
Problems and algorithms for covering arrays
Alan Hartman,Leonid Raskin +1 more
TL;DR: Several new problems motivated by covering arrays applications are raised and algorithms for their solution are discussed.
Proceedings ArticleDOI
A framework of greedy methods for constructing interaction test suites
TL;DR: A framework is developed to evaluate a large class of greedy methods that build suites one test at a time and provides a platform for optimizing the accuracy and speed of "one-test-at-a-time" greedy methods.