Prioritizing test cases for regression testing
TL;DR: Test case prioritization techniques schedule test cases for execution in an order that attempts to increase their effectiveness at meeting some performance goal as discussed by the authors, such as rate of fault detection, a measure of how quickly faults are detected within the testing process.
Abstract: Test case prioritization techniques schedule test cases for execution in an order that attempts to increase their effectiveness at meeting some performance goal. Various goals are possible; one involves rate of fault detection, a measure of how quickly faults are detected within the testing process. An improved rate of fault detection during testing can provide faster feedback on the system under test and let software engineers begin correcting faults earlier than might otherwise be possible. One application of prioritization techniques involves regression testing, the retesting of software following modifications; in this context, prioritization techniques can take advantage of information gathered about the previous execution of test cases to obtain test case orderings. We describe several techniques for using test execution information to prioritize test cases for regression testing, including: 1) techniques that order test cases based on their total coverage of code components; 2) techniques that order test cases based on their coverage of code components not previously covered; and 3) techniques that order test cases based on their estimated ability to reveal faults in the code components that they cover. We report the results of several experiments in which we applied these techniques to various test suites for various programs and measured the rates of fault detection achieved by the prioritized test suites, comparing those rates to the rates achieved by untreated, randomly ordered, and optimally ordered suites.
Citations
More filters
TL;DR: This paper surveys each area of minimization, selection and prioritization technique and discusses open problems and potential directions for future research.
Abstract: Regression testing is a testing activity that is performed to provide confidence that changes do not harm the existing behaviour of the software. Test suites tend to grow in size as software evolves, often making it too costly to execute entire test suites. A number of different approaches have been studied to maximize the value of the accrued test suite: minimization, selection and prioritization. Test suite minimization seeks to eliminate redundant test cases in order to reduce the number of tests to run. Test case selection seeks to identify the test cases that are relevant to some set of recent changes. Test case prioritization seeks to order test cases in such a way that early fault detection is maximized. This paper surveys each area of minimization, selection and prioritization technique and discusses open problems and potential directions for future research. Copyright © 2010 John Wiley & Sons, Ltd.
1,276 citations
TL;DR: The infrastructure that is being designed and constructed to support controlled experimentation with testing and regression testing techniques is described and the impact that this infrastructure has had and can be expected to have.
Abstract: Where the creation, understanding, and assessment of software testing and regression testing techniques are concerned, controlled experimentation is an indispensable research methodology. Obtaining the infrastructure necessary to support such experimentation, however, is difficult and expensive. As a result, progress in experimentation with testing techniques has been slow, and empirical data on the costs and effectiveness of techniques remains relatively scarce. To help address this problem, we have been designing and constructing infrastructure to support controlled experimentation with testing and regression testing techniques. This paper reports on the challenges faced by researchers experimenting with testing techniques, including those that inform the design of our infrastructure. The paper then describes the infrastructure that we are creating in response to these challenges, and that we are now making available to other researchers, and discusses the impact that this infrastructure has had and can be expected to have.
1,114 citations
19 May 2002
TL;DR: A new technique that uses color to visually map the participation of each program statement in the outcome of the execution of the program with a test suite, consisting of both passed and failed test cases is presented.
Abstract: One of the most expensive and time-consuming components of the debugging process is locating the errors or faults. To locate faults, developers must identify statements involved in failures and select suspicious statements that might contain faults. This paper presents a new technique that uses visualization to assist with these tasks. The technique uses color to visually map the participation of each program statement in the outcome of the execution of the program with a test suite, consisting of both passed and failed test cases. Based on this visual mapping, a user can inspect the statements in the program, identify statements involved in failures, and locate potentially faulty statements. The paper also describes a prototype tool that implements our technique along with a set of empirical studies that use the tool for evaluation of the technique. The empirical studies show that, for the subject we studied, the technique can be effective in helping a user locate faults in a program.
1,063 citations
TL;DR: In this article, the authors empirically compared the effectiveness of fine granularity and coarse granularity prioritization techniques using both controlled experiments and case studies, and found that the incorporation of measures of fault proneness into prioritization technique improves their effectiveness.
Abstract: To reduce the cost of regression testing, software testers may prioritize their test cases so that those which are more important, by some measure, are run earlier in the regression testing process. One potential goal of such prioritization is to increase a test suite's rate of fault detection. Previous work reported results of studies that showed that prioritization techniques can significantly improve rate of fault detection. Those studies, however, raised several additional questions: 1) Can prioritization techniques be effective when targeted at specific modified versions; 2) what trade-offs exist between fine granularity and coarse granularity prioritization techniques; 3) can the incorporation of measures of fault proneness into prioritization techniques improve their effectiveness? To address these questions, we have performed several new studies in which we empirically compared prioritization techniques using both controlled experiments and case studies.
880 citations
16 May 2009
TL;DR: A fully automated method for locating and repairing bugs in software that works on off-the-shelf legacy applications and does not require formal specifications, program annotations or special coding practices is introduced.
Abstract: Automatic program repair has been a longstanding goal in software engineering, yet debugging remains a largely manual process. We introduce a fully automated method for locating and repairing bugs in software. The approach works on off-the-shelf legacy applications and does not require formal specifications, program annotations or special coding practices. Once a program fault is discovered, an extended form of genetic programming is used to evolve program variants until one is found that both retains required functionality and also avoids the defect in question. Standard test cases are used to exercise the fault and to encode program requirements. After a successful repair has been discovered, it is minimized using structural differencing algorithms and delta debugging. We describe the proposed method and report experimental results demonstrating that it can successfully repair ten different C programs totaling 63,000 lines in under 200 seconds, on average.
722 citations
References
More filters
Book•
01 Jan 1979
TL;DR: The second edition of a quarterly column as discussed by the authors provides a continuing update to the list of problems (NP-complete and harder) presented by M. R. Garey and myself in our book "Computers and Intractability: A Guide to the Theory of NP-Completeness,” W. H. Freeman & Co., San Francisco, 1979.
Abstract: This is the second edition of a quarterly column the purpose of which is to provide a continuing update to the list of problems (NP-complete and harder) presented by M. R. Garey and myself in our book ‘‘Computers and Intractability: A Guide to the Theory of NP-Completeness,’’ W. H. Freeman & Co., San Francisco, 1979 (hereinafter referred to as ‘‘[G&J]’’; previous columns will be referred to by their dates). A background equivalent to that provided by [G&J] is assumed. Readers having results they would like mentioned (NP-hardness, PSPACE-hardness, polynomial-time-solvability, etc.), or open problems they would like publicized, should send them to David S. Johnson, Room 2C355, Bell Laboratories, Murray Hill, NJ 07974, including details, or at least sketches, of any new proofs (full papers are preferred). In the case of unpublished results, please state explicitly that you would like the results mentioned in the column. Comments and corrections are also welcome. For more details on the nature of the column and the form of desired submissions, see the December 1981 issue of this journal.
40,020 citations
Book•
01 Jan 1977
TL;DR: In this article, the Chi-square test of homogeneity of proportions is used to compare the proportions of different groups of individuals in a population to a single variable, and the Wilcoxon Signed-Rank Test is used for the comparison of different proportions.
Abstract: PART I: INTRODUCTION 1. WHAT IS STATISTICS? Introduction / Why Study Statistics? / Some Current Applications of Statistics / What Do Statisticians Do? / Quality and Process Improvement / A Note to the Student / Summary / Supplementary Exercises PART II: COLLECTING THE DATA 2. USING SURVEYS AND SCIENTIFIC STUDIES TO COLLECT DATA Introduction / Surveys / Scientific Studies / Observational Studies / Data Management: Preparing Data for Summarization and Analysis / Summary PART III: SUMMARIZING DATA 3. DATA DESCRIPTION Introduction / Describing Data on a Single Variable: Graphical Methods / Describing Data on a Single Variable: Measures of Central Tendency / Describing Data on a Single Variable: Measures of Variability / The Box Plot / Summarizing Data from More Than One Variable / Calculators, Computers, and Software Systems / Summary / Key Formulas / Supplementary Exercises PART IV: TOOLS AND CONCEPTS 4. PROBABILITY AND PROBABILITY DISTRIBUTIONS How Probability Can Be Used in Making Inferences / Finding the Probability of an Event / Basic Event Relations and Probability Laws / Conditional Probability and Independence / Bayes's Formula / Variables: Discrete and Continuous / Probability Distributions for Discrete Random Variables / A Useful Discrete Random Variable: The Binomial / Probability Distributions for Continuous Random Variables / A Useful Continuous Random Variable: The Normal Distribution / Random Sampling / Sampling Distributions / Normal Approximation to the Binomial / Summary / Key Formulas / Supplementary Exercises PART V: ANALYZING DATA: CENTRAL VALUES, VARIANCES, AND PROPORTIONS 5. INFERENCES ON A POPULATION CENTRAL VALUE Introduction and Case Study / Estimation of / Choosing the Sample Size for Estimating / A Statistical Test for / Choosing the Sample Size for Testing / The Level of Significance of a Statistical Test / Inferences about for Normal Population, s Unknown / Inferences about the Population Median / Summary / Key Formulas / Supplementary Exercises 6. COMPARING TWO POPULATION CENTRAL VALUES Introduction and Case Study / Inferences about 1 - 2: Independent Samples / A Nonparametric Alternative: The Wilcoxon Rank Sum Test / Inferences about 1 - 2: Paired Data / A Nonparametric Alternative: The Wilcoxon Signed-Rank Test / Choosing Sample Sizes for Inferences about 1 - 2 / Summary / Key Formulas / Supplementary Exercises 7. INFERENCES ABOUT POPULATION VARIANCES Introduction and Case Study / Estimation and Tests for a Population Variance / Estimation and Tests for Comparing Two Population Variances / Tests for Comparing k > 2 Population Variances / Summary / Key Formulas / Supplementary Exercises 8. INFERENCES ABOUT POPULATION CENTRAL VALUES Introduction and Case Study / A Statistical Test About More Than Two Population Variances / Checking on the Assumptions / Alternative When Assumptions are Violated: Transformations / A Nonparametric Alternative: The Kruskal-Wallis Test / Summary / Key Formulas / Supplementary Exercises 9. MULTIPLE COMPARISONS Introduction and Case Study / Planned Comparisons Among Treatments: Linear Contrasts / Which Error Rate Is Controlled / Multiple Comparisons with the Best Treatment / Comparison of Treatments to a Control / Pairwise Comparison on All Treatments / Summary / Key Formulas / Supplementary Exercises 10. CATEGORICAL DATA Introduction and Case Study / Inferences about a Population Proportion p / Comparing Two Population Proportions p1 - p2 / Probability Distributions for Discrete Random Variables / The Multinomial Experiment and Chi-Square Goodness-of-Fit Test / The Chi-Square Test of Homogeneity of Proportions / The Chi-Square Test of Independence of Two Nominal Level Variables / Fisher's Exact Test, a Permutation Test / Measures of Association / Combining Sets of Contingency Tables / Summary / Key Formulas / Supplementary Exercises PART VI: ANALYZING DATA: REGRESSION METHODS, MODEL BUILDING 11. SIMPLE LINEAR REGRESSION AND CORRELATION Linear Regression and the Method of Least Squares / Transformations to Linearize Data / Correlation / A Look Ahead: Multiple Regression / Summary of Key Formulas. Supplementary Exercises. 12. INFERENCES RELATED TO LINEAR REGRESSION AND CORRELATION Introduction and Case Study / Diagnostics for Detecting Violations of Model Conditions / Inferences about the Intercept and Slope of the Regression Line / Inferences about the Population Mean for a Specified Value of the Explanatory Variable / Predictions and Prediction Intervals / Examining Lack of Fit in the Model / The Inverse Regression Problem (Calibration): Predicting Values for x for a Specified Value of y / Summary / Key Formulas / Supplementary Exercises 13. MULTIPLE REGRESSION AND THE GENERAL LINEAR MODEL Introduction and Case Study / The General Linear Model / Least Squares Estimates of Parameters in the General Linear Model / Inferences about the Parameters in the General Linear Model / Inferences about the Population Mean and Predictions from the General Linear Model / Comparing the Slope of Several Regression Lines / Logistic Regression / Matrix Formulation of the General Linear Model / Summary / Key Formulas / Supplementary Exercises 14. BUILDING REGRESSION MODELS WITH DIAGNOSTICS Introduction and Case Study / Selecting the Variables (Step 1) / Model Formulation (Step 2) / Checking Model Conditions (Step 3) / Summary / Key Formulas / Supplementary Exercises PART VII: ANALYZING DATA: DESIGN OF EXPERIMENTS AND ANOVA 15. DESIGN CONCEPTS FOR EXPERIMENTS AND STUDIES Experiments, Treatments, Experimental Units, Blocking, Randomization, and Measurement Units / How Many Replications? / Studies for Comparing Means versus Studies for Comparing Variances / Summary / Key Formulas / Supplementary Exercises 16. ANALYSIS OF VARIANCE FOR STANDARD DESIGNS Introduction and Case Study / Completely Randomized Design with Single Factor / Randomized Block Design / Latin Square Design / Factorial Experiments in a Completely Randomized Design / The Estimation of Treatment Differences and Planned Comparisons in the Treatment Means / Checking Model Conditions / Alternative Analyses: Transformation and Friedman's Rank-Based Test / Summary / Key Formulas / Supplementary Exercises 17. ANALYSIS OF COVARIANCE Introduction and Case Study / A Completely Randomized Design with One Covariate / The Extrapolation Problem / Multiple Covariates and More Complicated Designs / Summary / Key Formulas / Supplementary Exercises 18. ANALYSIS OF VARIANCE FOR SOME UNBALANCED DESIGNS Introduction and Case Study / A Randomized Block Design with One or More Missing Observations / A Latin Square Design with Missing Data / Incomplete Block Designs / Summary / Key Formulas / Supplementary Exercises 19. ANALYSIS OF VARIANCE FOR SOME FIXED EFFECTS, RANDOM EFFECTS, AND MIXED EFFECTS MODELS Introduction and Case Study / A One-Factor Experiment with Random Treatment Effects / Extensions of Random-Effects Models / A Mixed Model: Experiments with Both Fixed and Random Treatment Effects / Models with Nested Factors / Rules for Obtaining Expected Mean Squares / Summary / Key Formulas / Supplementary Exercises 20. SPLIT-PLOT DESIGNS AND EXPERIMENTS WITH REPEATED MEASURES Introduction and Case Study / Split-Plot Designs / Single-Factor Experiments with Repeated Measures / Two-Factor Experiments with Repeated Measures on One of the Factors / Crossover Design / Summary / Key Formulas / Supplementary Exercises PART VIII: COMMUNICATING AND DOCUMENTING THE RESULTS OF A STUDY OR EXPERIMENT 21. COMMUNICATING AND DOCUMENTING THE RESULTS OF A STUDY OR EXPERIMENT Introduction / The Difficulty of Good Communication / Communication Hurdles: Graphical Distortions / Communication Hurdles: Biased Samples / Communication Hurdles: Sample Size / The Statistical Report / Documentation and Storage of Results / Summary / Supplementary Exercises
5,674 citations
TL;DR: In many cases tests of a program that uncover simple errors are also effective in uncovering much more complex errors, so-called coupling effect can be used to save work during the testing process.
Abstract: In many cases tests of a program that uncover simple errors are also effective in uncovering much more complex errors. This so-called coupling effect can be used to save work during the testing process.
2,047 citations
TL;DR: This paper defines a family of program test data selection criteria derived from data flow analysis techniques similar to those used in compiler optimization, arguing that currently used path selection criteria are inadequate.
Abstract: This paper defines a family of program test data selection criteria derived from data flow analysis techniques similar to those used in compiler optimization It is argued that currently used path selection criteria, which examine only the control flow of a program, are inadequate quate Our procedure associates with each point in a program at which a variable is defined, those points at which the value is used Several test data selection criteria, differing in the type and number of these associations, are defined and compared
1,084 citations