Fault coverage

About: Fault coverage is a research topic. Over the lifetime, 10153 publications have been published within this topic receiving 161933 citations. The topic is also known as: test coverage.

Do Automatically Generated Unit Tests Find Real Faults? An Empirical Study of Effectiveness and Challenges (T)

Abstract: Rather than tediously writing unit tests manually, tools can be used to generate them automatically --- sometimes even resulting in higher code coverage than manual testing. But how good are these tests at actually finding faults? To answer this question, we applied three state-of-the-art unit test generation tools for Java (Randoop, EvoSuite, and Agitar) to the 357 real faults in the Defects4J dataset and investigated how well the generated test suites perform at detecting these faults. Although the automatically generated test suites detected 55.7% of the faults overall, only 19.9% of all the individual test suites detected a fault. By studying the effectiveness and problems of the individual tools and the tests they generate, we derive insights to support the development of automated unit test generators that achieve a higher fault detection rate. These insights include 1) improving the obtained code coverage so that faulty statements are executed in the first instance, 2) improving the propagation of faulty program states to an observable output, coupled with the generation of more sensitive assertions, and 3) improving the simulation of the execution environment to detect faults that are dependent on external factors such as date and time.

Fault modeling and test algorithm development for static random access memories

Abstract: A fault model for SRAMs (static random-access memories) is presented based on physical spot defects, which are modeled as local disturbances in the layout of an SRAM. Two linear test algorithms (length 9N and 13N respectively, where N is the number of addresses) plus a data retention test are proposed that cover 100% of the faults under the fault model. The 13N test algorithm is generally applicable while the 9N algorithm can only be used in SRAMs with combinational R/W logic. A general solution is given for testing word-oriented SRAMs. The practical validity of the fault model and the two test algorithms is verified by a large number of actual wafer tests and device failure analysis. >

Comparison of physical and software-implemented fault injection techniques

Abstract: This paper addresses the issue of characterizing the respective impact of fault injection techniques. Three physical techniques and one software-implemented technique that have been used to assess the fault tolerance features of the MARS fault-tolerant distributed real-time system are compared and analyzed. After a short summary of the fault tolerance features of the MARS architecture and especially of the error detection mechanisms that were used to compare the erroneous behaviors induced by the fault injection techniques considered, we describe the common distributed testbed and test scenario implemented to perform a coherent set of fault injection campaigns. The main features of the four fault injection techniques considered are then briefly described and the results obtained are finally presented and discussed. Emphasis is put on the analysis of the specific impact and merit of each injection technique.

On Fault Representativeness of Software Fault Injection

Abstract: The injection of software faults in software components to assess the impact of these faults on other components or on the system as a whole, allowing the evaluation of fault tolerance, is relatively new compared to decades of research on hardware fault injection. This paper presents an extensive experimental study (more than 3.8 million individual experiments in three real systems) to evaluate the representativeness of faults injected by a state-of-the-art approach (G-SWFIT). Results show that a significant share (up to 72 percent) of injected faults cannot be considered representative of residual software faults as they are consistently detected by regression tests, and that the representativeness of injected faults is affected by the fault location within the system, resulting in different distributions of representative/nonrepresentative faults across files and functions. Therefore, we propose a new approach to refine the faultload by removing faults that are not representative of residual software faults. This filtering is essential to assure meaningful results and to reduce the cost (in terms of number of faults) of software fault injection campaigns in complex software. The proposed approach is based on classification algorithms, is fully automatic, and can be used for improving fault representativeness of existing software fault injection approaches.

Functional Testing of Semiconductor Random Access Memories

Abstract: This paper presents an overview of the problem of testing semiconductor random access memories (RAMs). An important aspect of this test procedure is the detection of permanent faults that cause the memory to function incorrectly. Functional-level fault models are very useful for describing a wide variety of RAM faults. Several fault models are &scussed throughout the paper, including the stuck-at-0/1 faults, coupled-cell faults, and single-cell pattern-sensitive faults. Test procedures for these fault models are presented and their fault coverage and execution times are discussed. The paper is intended for the general computer scmnce audience and presupposes no background in the hardware testing area.