Objective measurement of test quality is one of the key issues in software testing. It has been a major research focus for the last two decades. Many test criteria have been proposed and studied for this purpose. Various kinds of rationales have been presented in support of one criterion or another. We survey the research work in this area. The notion of adequacy criteria is examined together with its role in software dynamic testing. A review of criteria classification is followed by a summary of the methods for comparison and assessment of criteria.

/pdf/software-unit-test-coverage-and-adequacy-43qk91tobg.pdf

Software unit test coverage and adequacy

Regression testing is a testing activity that is performed to provide confidence that changes do not harm the existing behaviour of the software. Test suites tend to grow in size as software evolves, often making it too costly to execute entire test suites. A number of different approaches have been studied to maximize the value of the accrued test suite: minimization, selection and prioritization. Test suite minimization seeks to eliminate redundant test cases in order to reduce the number of tests to run. Test case selection seeks to identify the test cases that are relevant to some set of recent changes. Test case prioritization seeks to order test cases in such a way that early fault detection is maximized. This paper surveys each area of minimization, selection and prioritization technique and discusses open problems and potential directions for future research. Copyright © 2010 John Wiley & Sons, Ltd.

Regression testing minimization, selection and prioritization: a survey

Explicitly stated program invariants can help programmers by identifying program properties that must be preserved when modifying code. In practice, however, these invariants are usually implicit. An alternative to expecting programmers to fully annotate code with invariants is to automatically infer likely invariants from the program itself. This research focuses on dynamic techniques for discovering invariants from execution traces. This article reports three results. First, it describes techniques for dynamically discovering invariants, along with an implementation, named Daikon, that embodies these techniques. Second, it reports on the application of Daikon to two sets of target programs. In programs from Gries's work (1981) on program derivation, the system rediscovered predefined invariants. In a C program lacking explicit invariants, the system discovered invariants that assisted a software evolution task. These experiments demonstrate that, at least for small programs, invariant inference is both accurate and useful. Third, it analyzes scalability issues, such as invariant detection runtime and accuracy, as functions of test suites and program points instrumented.

/pdf/dynamically-discovering-likely-program-invariants-to-support-2hrrlyjk0k.pdf

Dynamically discovering likely program invariants to support program evolution

Test case prioritization techniques schedule test cases for execution in an order that attempts to increase their effectiveness at meeting some performance goal. Various goals are possible; one involves rate of fault detection, a measure of how quickly faults are detected within the testing process. An improved rate of fault detection during testing can provide faster feedback on the system under test and let software engineers begin correcting faults earlier than might otherwise be possible. One application of prioritization techniques involves regression testing, the retesting of software following modifications; in this context, prioritization techniques can take advantage of information gathered about the previous execution of test cases to obtain test case orderings. We describe several techniques for using test execution information to prioritize test cases for regression testing, including: 1) techniques that order test cases based on their total coverage of code components; 2) techniques that order test cases based on their coverage of code components not previously covered; and 3) techniques that order test cases based on their estimated ability to reveal faults in the code components that they cover. We report the results of several experiments in which we applied these techniques to various test suites for various programs and measured the rates of fault detection achieved by the prioritized test suites, comparing those rates to the rates achieved by untreated, randomly ordered, and optimally ordered suites.

Prioritizing test cases for regression testing

The high cost of locating faults in programs has motivated the development of techniques that assist in fault localization by automating part of the process of searching for faults. Empirical studies that compare these techniques have reported the relative effectiveness of four existing techniques on a set of subjects. These studies compare the rankings that the techniques compute for statements in the subject programs and the effectiveness of these rankings in locating the faults. However, it is unknown how these four techniques compare with Tarantula, another existing fault-localization technique, although this technique also provides a way to rank statements in terms of their suspiciousness. Thus, we performed a study to compare the Tarantula technique with the four techniques previously compared. This paper presents our study---it overviews the Tarantula technique along with the four other techniques studied, describes our experiment, and reports and discusses the results. Our studies show that, on the same set of subjects, the Tarantula technique consistently outperforms the other four techniques in terms of effectiveness in fault localization, and is comparable in efficiency to the least expensive of the other four techniques.

/pdf/empirical-evaluation-of-the-tarantula-automatic-fault-s84tf6v6nj.pdf

Empirical evaluation of the tarantula automatic fault-localization technique

Two different software fault prediction models have been used to predict the N% of the files of a large software system that are likely to contain the largest numbers of faults. We used the same predictor variables in a negative binomial regression model and a recursive partitioning model, and compared their effectiveness on three large industrial software systems. The negative binomial model identified files that contain 76 to 93 percent of the faults, and recursive partitioning identified files that contain 68 to 85 percent.

Comparing negative binomial and recursive partitioning models for fault prediction

Software testing research has not kept up with modern software system designs and applications, and software engineering education falls short of providing students with the type of knowledge and training that other engineering specialties require. Testing researchers should pay more attention to areas that are currently relevant for practicing software developers, such as embedded systems, mobile devices, safety-critical systems and other modern paradigms, in order to provide usable results and techniques for practitioners. We identify a number of skills that every software engineering student and faculty should have learned, and also propose that education for future software engineers should include significant exposure to real systems, preferably through hands-on training via internships at software-producing firms.

Software testing research and software engineering education

It is our great pleasure to welcome you to PROMISE 2008 - the 4th International Workshop on Predictor Models in Software Engineering. This year's workshop continues its tradition of being the premier forum for presentation of research results and experience reports in the area of predictor models applied to software engineering. The theme for this year's workshop is Bridging Research and Industry. Key questions for the workshop are -- How might PROMISE and other researchers better align with the realities of industry? -- How can industry make effective use of research ideas?

In keeping with this theme, we have two keynote speakers from industry. Dr. Murray Cantor is an IBM Distinguished Engineer and the governance solutions lead on the IBM Rational Software CTO team. Mr. Chris Beal is a Sun Microsystems Senior Staff Engineer working with Solaris Revenue Product Engineering.

We are pleased that PROMISE has become an international event. Our call for papers attracted submissions from Asia, Canada, Europe, and the United States. The program committee accepted over a dozen papers covering a variety of topics, including models related to fault prediction, effort estimation, and requirements engineering.

One feature that sets this workshop apart from others is the PROMISE repository of software data sets that are publicly available for research purposes. The repository currently has 57 data sets and has grown at an average rate of 44% annually over the last 3.5 years.

Proceedings of the 4th international workshop on Predictor models in software engineering

Previous studies have shown that software code attributes, such as lines of source code, and history information, such as the number of code changes and the number of faults in prior releases of software, are useful for predicting where faults will occur. In this study of two large industrial software systems, we investigate the effectiveness of adding information about calling structure to fault prediction models. Adding calling structure information to a model based solely on non-calling structure code attributes modestly improved prediction accuracy. However, the addition of calling structure information to a model that included both history and non-calling structure code attributes produced no improvement.

On the use of calling structure information to improve fault prediction

Industrial experience assessing the stability of a large mission-critical software project is reported. We observed that the project incurred significant additional delays in resolving the types of problems usually uncovered when assessing mission-critical software stability. We present plausible hypotheses about the possible causes of these additional delays.

Thomas J. Ostrand

Papers

Comparing negative binomial and recursive partitioning models for fault prediction

Software testing research and software engineering education

Proceedings of the 4th international workshop on Predictor models in software engineering

On the use of calling structure information to improve fault prediction

Experience Developing Software Using a Globally Distributed Workforce