Fault injection is important to evaluating the dependability of computer systems. Researchers and engineers have created many novel methods to inject faults, which can be implemented in both hardware and software. The contrast between the hardware and software methods lies mainly in the fault injection points they can access, the cost and the level of perturbation. Hardware methods can inject faults into chip pins and internal components, such as combinational circuits and registers that are not software-addressable. On the other hand, software methods are convenient for directly producing changes at the software-state level. Thus, we use hardware methods to evaluate low-level error detection and masking mechanisms, and software methods to test higher level mechanisms. Software methods are less expensive, but they also incur a higher perturbation overhead because they execute software on the target system.

/pdf/fault-injection-techniques-and-tools-67yktshngr.pdf

Fault injection techniques and tools

We present a study of operating system errors found by automatic, static, compiler analysis applied to the Linux and OpenBSD kernels. Our approach differs from previous studies that consider errors found by manual inspection of logs, testing, and surveys because static analysis is applied uniformly to the entire kernel source, though our approach necessarily considers a less comprehensive variety of errors than previous studies. In addition, automation allows us to track errors over multiple versions of the kernel source to estimate how long errors remain in the system before they are fixed.We found that device drivers have error rates up to three to seven times higher than the rest of the kernel. We found that the largest quartile of functions have error rates two to six times higher than the smallest quartile. We found that the newest quartile of files have error rates up to twice that of the oldest quartile, which provides evidence that code "hardens" over time. Finally, we found that bugs remain in the Linux kernel an average of 1.8 years before being fixed.

An empirical study of operating systems errors

Experimental evaluation

This paper focuses on the integration of the fault injection methodology within the design process of fault-tolerant systems. Due to its wide spectrum of application and hierarchical features, VHDL has been selected as the simulation language to support such an integration. Suitable techniques for injecting faults into VHDL models are identified and depicted. Then, the main features of the MEFISTO environment aimed at supporting these techniques are described. Finally, some preliminary results obtained with MEFISTO are presented and analyzed. >

Fault injection into VHDL models: the MEFISTO tool

An important step in the development of dependable systems is the validation of their fault tolerance properties Fault injection has been widely used for this purpose, however with the rapid increase in processor complexity, traditional techniques are also increasingly more difficult to apply This paper presents a new software-implemented fault injection and monitoring environment, called Xception, which is targeted at modern and complex processors Xception uses the advanced debugging and performance monitoring features existing in most modern processors to inject quite realistic faults by software, and to monitor the activation of the faults and their impact on the target system behavior in detail Faults are injected with minimum interference with the target application The target application is not modified, no software traps are inserted, and it is not necessary to execute the target application in special trace mode (the application is executed at full speed) Xception provides a comprehensive set of fault triggers, including spatial and temporal fault triggers, and triggers related to the manipulation of data in memory Faults injected by Xception can affect any process running on the target system (including the kernel), and it is possible to inject faults in applications for which the source code is not available Experimental, results are presented to demonstrate the accuracy and potential of Xception in the evaluation of the dependability properties of the complex computer systems available nowadays

Xception: a technique for the experimental evaluation of dependability in modern computers

A major step toward the development of fault-tolerant computer systems is the validation of the dependability properties of these systems. Fault/error injection has been recognized as a powerful approach to validate the fault tolerance mechanisms of a system and to obtain statistics on parameters such as coverages and latencies. This paper describes the methodology and guidelines for the design of flexible software based fault and error injection and presents a tool, FERRARI, that incorporates the techniques. The techniques used to emulate transient errors and permanent faults in software are described in detail. Experimental results are presented for several error detection techniques, and they demonstrate the effectiveness of the software-based error injection tool in evaluating the dependability properties of complex systems. >

FERRARI: a flexible software-based fault and error injection system

The authors present FERRARI, a fault and error automatic real-time injector, which can evaluate complex systems by emulating most hardware faults in software. The current version of FERRARI runs on SPARC workstations, in an Xwindow environment. The motivation, methodology, design, implementation, and evaluation of FERRARI are presented. The techniques used to emulate permanent faults and transient errors in software are described in detail. Experimental results are presented for several error detection techniques. They demonstrate the effectiveness of FERRARI in its role as a fault and error injector. >

FERRARI: a tool for the validation of system dependability properties

This paper presents a new hybrid fault/error injection technique which overcomes the limitations of both software-based and hardware-based approaches. The logic for the hardware fault unit injection circuitry is implemented using field programmable gate arrays, and the software is an extension of FERRARI, the software-based fault injection system. The combination of these techniques allows the incorporation of new capabilities by the use of mechanisms to trigger and synchronize the injection of a fault or error with events in the system. Results of physical fault/error injection experiments on a SPARC1 system are presented. The injection was synchronized to the executing modes and load conditions of the system. These results show that the system behavior is very sensitive to the internal state and load. Therefore, in order to validate the dependability properties of a system, it is imperative to inject faults/errors while the system is in critical conditions and different execution modes. >

Dependability evaluation using hybrid fault/error injection

Error detection and error isolation are becoming stringent requirements for many computational problems requiring high reliability in addition to high performance. This paper presents CAPACETI, a technique for utilizing capabilities at the application level in order to achieve dependable operations. The proposed technique is further augmented with executable assertions and other software error detection techniques. The effectiveness of the techniques to detect errors, their contribution to the overall coverage, and their performance overhead were experimentally obtained using fault/error injection. Results obtained from these experiments show that high coverage with a low performance overhead can be achieved by selectively combining different error detection techniques.

G.A. Kanawati

Papers

FERRARI: a flexible software-based fault and error injection system

FERRARI: a tool for the validation of system dependability properties

Dependability evaluation using hybrid fault/error injection

Adding capability checks enhances error detection and isolation in object-based systems