The results of several experiments conducted using the fault-injection-based automated testing (FIAT) system are presented. FIAT is capable of emulating a variety of distributed system architectures, and it provides the capabilities to monitor system behavior and inject faults for the purpose of experimental characterization and validation of a system's dependability. The experiments consists of exhaustively injecting three separate fault types into various locations, encompassing both the code and data portions of memory images, of two distinct applications executed with several different data values and sizes. Fault types are variations of memory bit faults. The results show that there are a limited number of system-level fault manifestations. These manifestations follow a normal distribution for each fault type. Error detection latencies are found to be normally distributed. The methodology can be used to predict the system-level fault responses during the system design stage. >

Fault injection experiments using FIAT

The authors describe a dependability evaluation method based on fault injection that establishes the link between the experimental evaluation of the fault tolerance process and the fault occurrence process. The main characteristics of a fault injection test sequence aimed at evaluating the coverage of the fault tolerance process are presented. Emphasis is given to the derivation of experimental measures. The various steps by which the fault occurrence and fault tolerance processes are combined to evaluate dependability measures are identified and their interactions are analyzed. The method is illustrated by an application to the dependability evaluation of the distributed fault-tolerant architecture of the Esprit Delta-4 Project. >

/pdf/fault-injection-and-dependability-evaluation-of-fault-3jtbahxygr.pdf

Fault injection and dependability evaluation of fault-tolerant systems

The paper presents the rationale for a functional simulation tool, called DEPEND, which provides an integrated design and fault injection environment for system level dependability analysis. The paper discusses the issues and problems of developing such a tool, and describes how DEPEND tackles them. Techniques developed to simulate realistic fault scenarios, reduce simulation time explosion, and handle the large fault model and component domain associated with system level analysis are presented. Examples are used to motivate and illustrate the benefits of this tool. To further illustrate its capabilities, DEPEND is used to simulate the Unix-based Tandem triple-modular-redundancy (TMR) based prototype fault-tolerant system and to evaluate how well it handles near-coincident errors caused by correlated and latent faults. Issues such as memory scrubbing, re-integration policies, and workload dependent repair times, which affect how the system handles near-coincident errors, are also evaluated. Unlike any other simulation-based dependability studies, the accuracy of the simulation model is validated by comparing the results of the simulations with measurements obtained from fault injection experiments conducted on a production Tandem machine.

/pdf/depend-a-simulation-based-environment-for-system-level-511jigws3x.pdf

DEPEND: a simulation-based environment for system level dependability analysis

Designers are realizing the advantages of performing fault injection early, using simulation to inject faults into a model of the design rather than the actual system. The authors describe their technique for injecting faults into a system's VHDL behavioral level model. To demonstrate the technique, they evaluate an embedded control system providing fail safe operation in the railway industry.

A fault injection technique for VHDL behavioral-level models

This paper demonstrates a practical methodology for the study of error latency under a real workload. The method is illustrated with sampled data on the physical memory activity, gathered by hardware instrumentation on a VAX 11/780 during the normal workload cycle of the installation. These data are used to simulate fault occurrence and to reconstruct the error discovery process in the system. The technique provides a means to study the system under different workloads and for multiple days. An approach to determine the percentage of undiscovered errors is also developed and a verification of the entire methodology is performed. This study finds that the mean error latency, in the memory containing the operating system, varies by a factor of 10 to 1 (in hours) between the low and high workloads. It is found that of all errors occurring within a day, 70 percent are detected in the same day, 82 percent within the following day, and 91 percent within the third day. The increase in failure rate due to latency is not so much a function of remaining errors but is dependent on whether or not there is a latent error.

Measurement-Based Analysis of Error Latency

Studies carried out by McGough and Swern (1981, 1983) are summarized. In these studies, an avionics processor was simulated and a series of fault injection experiments was carried out to determine the degree of fault latency in a redundant flight control system that employed comparison monitoring as the exclusive means of failure detection. A determination was also made of the fault coverage of a typical self-test program. The summary presented stresses that a self-test program should be designed to capitalize on the hardware mechanization of the processor. If this is not done, subtests tend to repeatedly exercise the same hardware components while neglecting to exercise a substantial proportion of the remainder. It is also pointed out that fault latency is relatively independent of both the length and instruction mix of a program. A significant difference is found in fault coverage assessed using pin-level and gate-level fault models.

New results in fault latency modelling

Using a gate level emulation of a typical avionics miniprocessor, fault injection experiments were performed to (1) determine the time to detect a fault by comparison monitoring, (2) forecast a program's ability to detect faults and (3) validate the fault detection coverage of a typical self test program. To estimate time to detect, six programs ranging in complexity from 6 to 147 instructions, were emulated. Each program was executed repetitively in the presence of a single stuck at fault at a gate node or device pin. Detection was assumed to occur whenever the computed outputs differed from the corresponding outputs of the same program executed in a nonfaulted processor. Histograms of faults detected versus number of repetitions to detection were tabulated. Using a simple model of fault detection, which was based in an analog with the selection of balls in an urn, distributions of time to detect were computed and compared with those obtained empirically. A self test program of 2,000 executable instructions was designed expressly for the study. The only requirement imposed on the design was that it should achieve 95% coverage. The program was executed in the presence of a single stuck-at fault at a gate node on device pin. The proportion of detected faults are tabulated. In all experiments faults were selected at random over gate nodes or device pins.

J. G. Mcgough

Papers

New results in fault latency modelling

Methodology for measurement of fault latency in a digital avionic miniprocessor