Fail-stop failures in distributed environments are often tolerated by checkpointing or message logging. In this paper, we show that fail-stop process failures in ScaLAPACK matrix-matrix multiplication kennel can be tolerated without checkpointing or message logging. It has been proved in previous algorithm-based fault tolerance that, for matrix-matrix multiplication, the checksum relationship in the input checksum matrices is preserved at the end of the computation no mater which algorithm is chosen. From this checksum relationship in the final computation results, processor miscalculations can be detected, located, and corrected at the end of the computation. However, whether this checksum relationship can be maintained in the middle of the computation or not remains open. In this paper, we first demonstrate that, for many matrix matrix multiplication algorithms, the checksum relationship in the input checksum matrices is not maintained in the middle of the computation. We then prove that, however, for the outer product version algorithm, the checksum relationship in the input checksum matrices can be maintained in the middle of the computation. Based on this checksum relationship maintained in the middle of the computation, we demonstrate that fail-stop process failures (which are often tolerated by checkpointing or message logging) in ScaLAPACK matrix-matrix multiplication can be tolerated without checkpointing or message logging.

/pdf/algorithm-based-fault-tolerance-for-fail-stop-failures-25pod6j3h7.pdf

Algorithm-Based Fault Tolerance for Fail-Stop Failures

This paper presents several techniques for tolerating faults in d-dimensional mesh and hypercube architectures. The approach consists of adding spare processors and communication links so that the resulting architecture will contain a fault-free mesh or hypercube in the presence of faults. The authors optimize the cost of the fault-tolerant architecture by adding exactly k spare processors (while tolerating up to k processor and/or link faults) and minimizing the maximum number of links per processor. For example, when the desired architecture is a d-dimensional mesh and k=1, they present a fault-tolerant architecture that has the same maximum degree as the desired architecture (namely, 2d) and has only one spare processor. They also present efficient layouts for fault-tolerant two- and three-dimensional meshes, and show how multiplexers and buses can be used to reduce the degree of fault-tolerant architectures. Finally, they give constructions for fault-tolerant tori, eight-connected meshes, and hexagonal meshes. >

/pdf/fault-tolerant-meshes-and-hypercubes-with-minimal-numbers-of-2pivgdkmtu.pdf

Fault-tolerant meshes and hypercubes with minimal numbers of spares

As the size of today's high performance computers increases from hundreds, to thousands, and even tens of thousands of processors, node failures in these computers are becoming frequent events. Although checkpoint/rollback-recovery is the typical technique to tolerate such failures, it often introduces a considerable overhead. Algorithm-based fault tolerance is a very cost-effective method to incorporate fault tolerance into matrix computations. However, previous algorithm-based fault tolerance methods for matrix computations are often derived using algorithms that are seldomly used in the practice of today's high performance matrix computations and have mostly focused on platforms where failed processors produce incorrect calculations. To fill this gap, this paper extends the existing algorithm-based fault tolerance to the volatile computing platform where the failed processor stops working and applies it to scalable high performance matrix computations with two dimensional block cyclic data distribution. We show the practicality of this technique by applying it to the ScaLAPACK/PBLAS matrix-matrix multiplication kernel. Experimental results demonstrate that the proposed approach is able to survive process failures with a very low performance overhead.

/pdf/algorithm-based-checkpoint-free-fault-tolerance-for-parallel-cxjoie56zy.pdf

Algorithm-based checkpoint-free fault tolerance for parallel matrix computations on volatile resources

A theory for error detection in linear digital state variable systems is described. With the aid of a tool called the gain matrix, it is shown that the effect of error propagation along different paths of the circuit can be analyzed. For circuits with operator fanout, it is shown that despite the fact that single faulty operators cause multiple state variables to be erroneous, no more additional check variables are required than for circuits without operator fanout. It is further shown that hardware optimization can be performed by sharing hardware functions between the original state variable system and its error detection circuitry. The analysis is performed for both single and multiple faulty operators. A scheme for error correction that performs error correction in real time is proposed. Experimental results that illustrate the practical viability of the proposed scheme are discussed. >

The design of fault-tolerant linear digital state variable systems: theory and techniques

A critical component of any large-scale parallel processing system is the interconnection network that provides a means for communication along the system's processors and memories. Attributes of the multistage cube topology that have made it an effective basis for interconnection networks and the subject of much ongoing research are reviewed. These attributes include O(N log/sub 2/N) cost for an N-input/output network, decentralized control, a variety of implementation options, good data-permuting capability to support single-instruction-stream/multiple-data-stream (SIMD) parallelism, good throughput to support multiple-instruction-stream/multiple-data-stream (MIMD) parallelism, and ability to be partitioned into independent subnetworks to support reconfigurable systems. Examples of existing systems that use multistage cube networks are considered. The multistage cube topology can be converted into a single-stage network by associating with each switch in the network a processor (and a memory). Properties of systems that use the multistage cube network in this way are examined. >

Using the multistage cube network topology in parallel supercomputers

The task of synthesizing algorithm-based checking techniques for general applications is investigated. The problem is approached at the compiler level by identifying linear transformations in Fortran DO loops and restructuring program statements to convert nonlinear transformations to linear ones. System-level checks based on this property are proposed. The approach is demonstrated with example problems of matrix multiplication and the LINPACK routine: DGEFA. >

Compiler-assisted synthesis of algorithm-based checking in multiprocessors

A fault tolerant massively parallel processing architecture

Numerous algorithms for computationally intensive tasks that are suitable for execution on hypercube multiprocessors have been developed by researchers. In this thesis, we look at parallel algorithm design from a different perspective: the provision of on-line detection of hardware errors using software techniques without any hardware modifications. This approach is called Algorithm-based error detection. We report on the implementation of system-level error detection mechanisms for four parallel applications on a 16-processor Intel iPSC-2/D4/MX hypercube multiprocessor: (1) matrix multiplication; (2) Fast Fourier Transform; (3) QR factorization; (4) singular value decomposition. We describe extensive studies of the error coverage of our system-level error detection schemes in the presence of finite precision arithmetic which affects our system-level encodings.
We also provide an in-depth study of the various issues and trade-offs available in Algorithm-based error detection. We illustrate the approach on an extremely useful computation in the field of numerical linear algebra: QR factorization. We discuss the implementation and investigation of numerous ways of applying Algorithm-based error detection using different system-level encoding strategies for QR factorization. Different schemes have been observed to result in varying error coverages and time overheads. We report the result of our studies performed on the Intel iPSC-2 hypercube.
Finally, we address the difficult task of synthesizing algorithm-based checking techniques for general applications. We approach the problem at the compiler level by identifying linear transformations in Fortran DO loops, restructuring program statements to convert nonlinear transformations to linear ones, and propose system-level checks based on this property. We discuss the implementation of a source-to-source restructuring compiler for the synthesis of low-cost system-level checks for general numerical Fortran programs, based on the above approach. We present the results of the application of this compiler to various routines from the LINPACK and EISPACK libraries, and from the Perfect Benchmark Suite. We also provide a detailed evaluation of compiler-assisted techniques for the LINPACK routine, DGEFA, and demonstrate the feasibility of our concept experimentally on the Intel iPSC-2 hypercube.

The analysis and synthesis of efficient algorithm-based error detection schemes for hypercube multiprocessors

In many cases, it may be possible to redesign parallel algorithms so as to provide a low-cost online scheme for hardware error detection without any hardware modifications. This approach is called algorithm-based error detection. Two useful computations in signal processing are analyzed: QR factorization and singular-value decomposition. For each of these applications, numerous ways of applying algorithm-based error detection using different system-level encoding strategies are investigated. Different schemes have been observed to result in varying error coverages and time overheads. The results of studies performed on a 16-processor Intel iPSC-2/D4/MX hypercube multiprocessor are reported. >

Vijay Balasubramanian

Papers

Compiler-assisted synthesis of algorithm-based checking in multiprocessors

A fault tolerant massively parallel processing architecture

The analysis and synthesis of efficient algorithm-based error detection schemes for hypercube multiprocessors

Algorithm-based error detection for signal processing applications on a hypercube multiprocessor

RECBAR : A Reconfigurable Massively Parallel Processing Architecture.