Experimental evaluation

A fault tolerant computer system's dependability must be validated to ensure that its redundancy has been correctly implemented and the system will provide the desired level of reliable service. Fault injection-the deliberate insertion of faults into an operational system to determine its response offers an effective solution to this problem. We survey several fault injection studies and discuss tools such as React (Reliable Architecture Characterization Tool) that facilitate its application. >

Fault injection: a method for validating computer-system dependability

This paper reviews an area which has evolved over the past 15 years: experimental analysis of computer system dependability. Methodologies and advances are discussed for three basic approaches used in the area: simulated fault injection, physical fault injection, and measurement-based analysis. The three approaches are suited, respectively, to dependability evaluation in the three phases of a system's life: design phase, prototype phase, and operational phase. Before the discussion of these phases, several statistical techniques used in the area are introduced. For each phase, a classification of research methods or study topics is outlined, followed by discussion of these methods or topics as well as representative studies. The statistical techniques introduced include the estimation of parameters and confidence intervals, probability distribution characterization, and several multivariate analysis methods. Importance sampling, a statistical technique used to accelerate Monte Carlo simulation, is also introduced. The discussion of simulated fault injection covers electrical-level, logic-level, and function-level fault injection methods as well as representative simulation environments such as FOCUS and DEPEND. The discussion of physical fault injection covers hardware, software, and radiation fault injection methods as well as several software and hybrid tools including FIAT, FERARI, HYBRID, and FINE. The discussion of measurement-based analysis covers measurement and data processing techniques, basic error characterization, dependency analysis, Markov reward modeling, software-dependability, and fault diagnosis. The discussion involves several important issues studies in the area, including fault models, fast simulation techniques, workload/failure dependency, correlated failures, and software fault tolerance.

Experimental analysis of computer system dependability

A simulation-based software-model that permits application specific dependability analysis in the early design stages is introduced. The model represents an application program by decomposing it into a graph model consisting of a set of nodes, a set of edges that probabilistically determine the flow from node to node, and a mapping of the nodes to memory. The software model simulates the execution of the program while errors are injected into the program's memory space. The model provides application-dependent parameters such as detection and propagation times and permits evaluation of function on system level error detection and recovery schemes. A case study illustrates the interaction between an application program and two detection schemes. Specifically, Gaussian elimination programs running on a Tandem Integrity S2 system with memory scrubbing are studied. Results from the simulation-based software model are validated with data measured from an actual Tandem Integrity S2 system. Application dependent coverage values obtained with the model are compared with those obtained via traditional schemes that assume uniform or ramp memory access patterns. For the authors' program, some coverage values obtained with the traditional approaches were found to be 100% larger than those obtained with the software model.

Simulation of Software Behavior Under Hardware Faults

This paper presents a hierarchical simulation methodology that enables accurate system evaluation under realistic faults and conditions. In this methodology, effects of low-level (i.e., transistor or circuit level) faults are propagated to higher levels (i.e., system level) using fault dictionaries. The primary fault models are obtained via simulation of the transistor-level effect of a radiation particle penetrating a device. The resulting current bursts constitute the first-level fault dictionary and are used in the circuit-level simulation to determine the impact on circuit latches and flip-flops. The latched outputs constitute the next level fault dictionary in the hierarchy and are applied in conducting fault injection simulation at the chip-level under selected workloads or application programs. Faults injected at the chip-level result in memory corruptions, which are used to form the next level fault dictionary for the system-level simulation of an application running on simulated hardware. When an application terminates, either normally or abnormally, the overall fault impact on the software behavior is quantified and analyzed. The system in this sense can be a single workstation or a network. The simulation method is demonstrated and validated in the case study of Myrinet (a commercial, high-speed network) based network system.

Hierarchical simulation approach to accurate fault modeling for system dependability evaluation

A software testbed that performs automated life testing of a variety of multiprocessor architectures through simulated fault injection is discussed. It is being developed to meet the need for a generalized simulation tool which can evaluate system reliability and availability metrics while avoiding several of the limitations associated with combinatorial and Markov modeling. Incorporating detailed system, workload, and fault/error models, REACT can be more accurate and easier to use than many dependability prediction tools based on analytical approaches. The authors motivate the development of REACT, describe its features, and explain its use. Example applications for the software are provided, and its limitations are discussed. >

http://ieeexplore.ieee.org/iel3/1050/7344/00296819.pdf

D.K. Pradhan

Papers

REACT: a synthesis and evaluation tool for fault-tolerant multiprocessor architectures