Many software testing fields, like white-box testing, test case generation, test prioritization, and fault localization, depend on code coverage measurement. If used as an overall completeness measure, the minor inaccuracies of coverage data reported by a tool do not matter that much; however, in certain situations, they can lead to serious confusion. For example, a code element that is falsely reported as covered can introduce false confidence in the test. This work investigates code coverage measurement issues for the Java programming language. For Java, the prevalent approach to code coverage measurement is using bytecode instrumentation due to its various benefits over source code instrumentation. As we have experienced, bytecode instrumentation-based code coverage tools produce different results than source code instrumentation-based ones in terms of the reported items as covered. We report on an empirical study to compare the code coverage results provided by tools using the different instrumentation types for Java coverage measurement on the method level. In particular, we want to find out how much a bytecode instrumentation approach is inaccurate compared to a source code instrumentation method. The differences are systematically investigated both in quantitative (how much the outputs differ) and in qualitative terms (what the causes for the differences are). In addition, the impact on test prioritization and test suite reduction—a possible application of coverage measurement—is investigated in more detail as well.

/pdf/code-coverage-differences-of-java-bytecode-and-source-code-si6b04pn8f.pdf

Code coverage differences of Java bytecode and source code instrumentation tools

Code coverage measurement is an important element in white-box testing, both in industrial practice and academic research. Other related areas are highly dependent on code coverage as well, including test case generation, test prioritization, fault localization, and others. Inaccuracies of a code coverage tool sometimes do not matter that much but in certain situations they can lead to serious confusion. For Java, the prevalent approach to code coverage measurement is to use bytecode instrumentation due to its various benefits over source code instrumentation. However, if the results are to be mapped back to source code this may lead to inaccuracies due to the differences between the two program representations. In this paper, we systematically investigate the amount of differences in the results of these two Java code coverage approaches, enumerate the possible reasons and discuss the implications on various applications. For this purpose, we relied on two widely used tools to represent the two approaches and a set of benchmark programs from the open source domain.

/pdf/negative-effects-of-bytecode-instrumentation-on-java-source-4tmwvxmqkv.pdf

Negative Effects of Bytecode Instrumentation on Java Source Code Coverage

Software Testing: Tools

Code coverage is one of the most important aspects of software testing, which helps software engineers to understand which portion of code has been executed using test suite throughout the software testing process. Automatic testing tools are widely used to provide testing coverage metrics in order to gauge the quality of software, but these tools may suffer from some shortcomings such as the difference among the values of code coverage metric of a given program using different code coverage tools. Therefore, we designed and performed a controlled experiment to investigate whether these tools have a significant difference among the measured values of coverage metric or not. We collected the coverage data that consist of branch, line, statement, and method coverage metrics. Statistically, our findings show that there is a significant difference of code coverage results among the code coverage tools in terms of branch and method coverage metrics.

Evaluation of Software Testing Coverage Tools: An Empirical Study

Competencer wants to be sure users experience their web product as it was designed. With the help of tools for end-to-end user testing, interactions based on what a user would do is simulated to test potential situations.This thesis work is targeting four areas of end-to-end user testing with a major focus on making it automatic. A study is conducted on test case methods to gain an understanding of how to approach writing tests. A coverage tool is researched and built to present a measure of what is being tested of the product. To ease the use for developers a solution for continuous integration is looked at. To make tests more automatic a way to build mocks through automation is implemented.These areas combined with the background of Competencers application architecture creates a foundation for replacing manual testing sessions with automatic ones.

/pdf/automated-end-to-end-user-testing-on-single-page-web-3rhduvy9ln.pdf

Automated end-to-end user testing on single page web applications

coverage metric is considered as the most important metric used in analysis of software projects for testing. Code coverage analysis also helps in the testing process by finding areas of a program not exercised by a set of test cases, creating additional test cases to increase coverage, and determine the quantitative measure of the code, which is an indirect measure of quality. There are a large number of automated tools to find the coverage of test cases in Java. Choosing an appropriate tool for the application to be tested may be a complicated process. To make it ease we propose an approach for measuring characteristics of these testing tools in order to evaluate them systematically and to select the appropriate one. Keywordscoverage metrics, testing tools, test case, test suite

/pdf/an-evaluation-of-java-code-coverage-testing-tools-2tsrv9hce3.pdf

Megi Tartari

Papers

An Evaluation of Java Code Coverage Testing Tools