scispace - formally typeset
SciSpace - Your AI assistant to discover and understand research papers | Product Hunt

Journal ArticleDOI

Estimating software testing complexity

01 Dec 2013-Information & Software Technology (Butterworth-Heinemann)-Vol. 55, Iss: 12, pp 2125-2139

TL;DR: The definition of a new measure of the difficulty for a computer to generate test cases, called Branch Coverage Expectation (BCE), which is much more correlated with the code coverage than the existing complexity measures.

AbstractContext: Complexity measures provide us some information about software artifacts. A measure of the difficulty of testing a piece of code could be very useful to take control about the test phase. Objective: The aim in this paper is the definition of a new measure of the difficulty for a computer to generate test cases, we call it Branch Coverage Expectation (BCE). We also analyze the most common complexity measures and the most important features of a program. With this analysis we are trying to discover whether there exists a relationship between them and the code coverage of an automatically generated test suite. Method: The definition of this measure is based on a Markov model of the program. This model is used not only to compute the BCE, but also to provide an estimation of the number of test cases needed to reach a given coverage level in the program. In order to check our proposal, we perform a theoretical validation and we carry out an empirical validation study using 2600 test programs. Results: The results show that the previously existing measures are not so useful to estimate the difficulty of testing a program, because they are not highly correlated with the code coverage. Our proposed measure is much more correlated with the code coverage than the existing complexity measures. Conclusion: The high correlation of our measure with the code coverage suggests that the BCE measure is a very promising way of measuring the difficulty to automatically test a program. Our proposed measure is useful for predicting the behavior of an automatic test case generator.

Topics: Code coverage (72%), Test suite (62%), Test data generation (62%), Test case (61%), Dynamic program analysis (60%)

...read more

Content maybe subject to copyright    Report

Citations
More filters


Journal ArticleDOI
TL;DR: Fine-grained problem handling reports provide a wider scope of possible measures to assess the relevant development processes and provide significant enhancement in evaluation and refinement of software development processes as well as in reliability prediction.
Abstract: Context Although many papers have been published on software development and defect prediction techniques, problem reports in real projects quite often differ from those described in the literature. Hence, there is still a need for deeper exploration of case studies from industry. Objective The aim of this study is to present the impact of fine-grained problem reports on improving evaluation of testing and maintenance processes. It is targeted at projects involving several releases and complex schemes of problem handling. This is based on our experience gained while monitoring several commercial projects. Method Extracting certain features from detailed problem reports, we derive various measures and present analysis models which characterize and visualize the effectiveness of testing and problem resolution processes. The considered reports describe types of problems (e.g. defects), their locations in project versions and software modules, ways of their resolution, etc. The performed analysis is related to eleven projects developed in the same company. This study is an exploratory research with some explanatory features. Moreover, having identified some drawbacks, we present extensions of problem reports and their analysis which have been verified in another industrial case study project. Results Fine-grained (accurate) problem handling reports provide a wider scope of possible measures to assess the relevant development processes. This is helpful in controlling single projects (local perspective) as well as in managing these processes in the whole company (global perspective). Conclusion Detailed problem handling reports extend the space and quality of statistical analysis, they provide significant enhancement in evaluation and refinement of software development processes as well as in reliability prediction.

17 citations


Journal ArticleDOI
TL;DR: It is argued that knowing a priori the branch coverage that can be achieved with test‐data generation tools can help developers into taking informed decision about issues and it is investigated the possibility to use source‐code metrics to predict the coverage achieved by test‐ data generation tools.
Abstract: This is the peer reviewed version which has been published in final form at [DOI]. This article may be used for non-commercial purposes in accordance with Wiley Terms and Conditions for Use of Self-Archived Versions.

16 citations


Proceedings ArticleDOI
20 Mar 2018
TL;DR: The first steps towards the definition of machine learning models to predict the branch coverage achieved by test data generation tools are taken, considering well known code metrics as a features.
Abstract: Software testing is a crucial component in modern continuous integration development environment. Ideally, at every commit, all the system's test cases should be executed and moreover, new test cases should be generated for the new code. This is especially true in a Continuous Test Generation (CTG) environment, where the automatic generation of test cases is integrated into the continuous integration pipeline. Furthermore, developers want to achieve a minimum level of coverage for every build of their systems. Since both executing all the test cases and generating new ones for all the classes at every commit is not feasible, they have to select which subset of classes has to be tested. In this context, knowing a priori the branch coverage that can be achieved with test data generation tools might give some useful indications for answering such a question. In this paper, we take the first steps towards the definition of machine learning models to predict the branch coverage achieved by test data generation tools. We conduct a preliminary study considering well known code metrics as a features. Despite the simplicity of these features, our results show that using machine learning to predict branch coverage in automated testing is a viable and feasible option.

13 citations


Proceedings ArticleDOI
01 Jul 2017
TL;DR: This paper proposes an adaptive fitness function based on branch hardness, evaluated by the variation coefficient of the fitness function, of generating test data can be minimized and is more flexible than the traditional counterparts.
Abstract: Search based software testing has received great attention as a means of automating the test data generation, and the goal is to improve various criteria. There are different types of coverage criteria. In this paper, we deal with the path coverage. Concretely, we focus on the path that is the most difficult to cover. One major limitation of search based testing is the inefficient and insufficiently informed fitness function. To address this problem, we propose an adaptive fitness function based on branch hardness. The branch hardness is measured by the expected number of visits of each branch in the program, which is modeled by an absorbing discrete time Markov chain. By tuning the parameters of branch hardness heuristically, the search hardness, evaluated by the variation coefficient of the fitness function, of generating test data can be minimized. Therefore, this new fitness function is more flexible than the traditional counterparts. In addition, we point out that the present definition of branch distance and the use of normalizing functions are problematic, and propose some improvements. Finally, the empirical study reveals the promising result of our proposal in this paper.

12 citations


References
More filters

Book
04 Oct 1993
Abstract: This paper describes a graph-theoretic complexity measure and illustrates how it can be used to manage and control program complexity. The paper first explains how the graph theory concepts apply and gives an intuitive explanation of the graph concepts in programming terms. The control graphs of several actual FORTRAN programs are then presented to illustrate the correlation between intuitive complexity and the graph theoretic complexity. Several properties of the graph-theoretic complexity are then proved which show, for example, that complexity is independent of physical size (adding or subtracting functional statements leaves complexity unchanged) and complexity depends only on the decision structure of a program.The issue of using non-structured control flow is also discussed. A characterization of non-structured control graphs is given and a method of measuring the “structuredness” of a program is developed. The relationship between structure and reducibility is illustrated with several examples.The last section of the paper deals with a testing methodology used in conjunction with the complexity measure; a testing strategy is defined that dictates that a program can either admit of a certain minimal testing level or the program can be structurally reduced.

5,171 citations


Journal ArticleDOI
TL;DR: Several properties of the graph-theoretic complexity are proved which show, for example, that complexity is independent of physical size and complexity depends only on the decision structure of a program.
Abstract: This paper describes a graph-theoretic complexity measure and illustrates how it can be used to manage and control program complexity. The paper first explains how the graph-theory concepts apply and gives an intuitive explanation of the graph concepts in programming terms. The control graphs of several actual Fortran programs are then presented to illustrate the correlation between intuitive complexity and the graph-theoretic complexity. Several properties of the graph-theoretic complexity are then proved which show, for example, that complexity is independent of physical size (adding or subtracting functional statements leaves complexity unchanged) and complexity depends only on the decision structure of a program.

4,749 citations


"Estimating software testing complex..." refers background in this paper

  • ...This measure determines the minimum number of test cases needed to test all the paths using linearly independent circuits [25]....

    [...]


Book
01 Mar 1979
TL;DR: Comprehensively covers psychological and economic principles, managerial aspects of testing, test tools, high-order testing, code inspections, and debugging, and programming students will find this reference work indispensible.
Abstract: From the Publisher: Provides a practical rather than theoretical discussion of the purpose and nature of software testing. Emphasizes methodologies for the design of effective test cases. Comprehensively covers psychological and economic principles, managerial aspects of testing, test tools, high-order testing, code inspections, and debugging. Extensive bibliography. Programmers at all levels, and programming students, will find this reference work indispensible.

3,725 citations


"Estimating software testing complex..." refers background in this paper

  • ...Besides that, it is estimated that half the time spent on the software project development and more than half its cost, is devoted to testing the product [27]....

    [...]


Book
01 Jan 1979
Abstract: The classic, landmark work on software testingThe hardware and software of computing have changed markedly in the three decades since the first edition of The Art of Software Testing, but this book's powerful underlying analysis has stood the test of time. Whereas most books on software testing target particular development techniques, languages, or testing methods, The Art of Software Testing, Third Edition provides a brief but powerful and comprehensive presentation of time-proven software testing approaches. If your software development project is mission critical, this book is an investment that will pay for itself with the first bug you find.The new Third Edition explains how to apply the book's classic principles to today's hot topics including:Testing apps for iPhones, iPads, BlackBerrys, Androids, and other mobile devicesCollaborative (user) programming and testingTesting for Internet applications, e-commerce, and agile programming environmentsWhether you're a student looking for a testing guide you'll use for the rest of your career, or an IT manager overseeing a software development team, The Art of Software Testing, Third Edition is an expensive book that will pay for itself many times over.

3,313 citations


Book
01 Jan 1997
TL;DR: The Oxford University Press and the Institute of Physics have joined forces to create a major reference publication devoted to EC fundamentals, models, algorithms and applications, intended to become the standard reference resource for the evolutionary computation community.
Abstract: From the Publisher: Many scientists and engineers now use the paradigms of evolutionary computation (genetic agorithms, evolution strategies, evolutionary programming, genetic programming, classifier systems, and combinations or hybrids thereof) to tackle problems that are either intractable or unrealistically time consuming to solve through traditional computational strategies Recently there have been vigorous initiatives to promote cross-fertilization between the EC paradigms, and also to combine these paradigms with other approaches such as neural networks to create hybrid systems with enhanced capabilities To address the need for speedy dissemination of new ideas in these fields, and also to assist in cross-disciplinary communications and understanding, Oxford University Press and the Institute of Physics have joined forces to create a major reference publication devoted to EC fundamentals, models, algorithms and applications This work is intended to become the standard reference resource for the evolutionary computation community The Handbook of Evolutionary Computation will be available in loose-leaf print form, as well as in an electronic version that combines both CD-ROM and on-line (World Wide Web) acess to its contents Regularly published supplements will be available on a subscription basis

2,434 citations