Increase in graphics hardware performance and improvements in programmability has enabled GPUs to evolve from a graphics-specific accelerator to a general-purpose computing device. Titan, the world's second fastest supercomputer for open science in 2014, consists of more dum 18,000 GPUs that scientists from various domains such as astrophysics, fusion, climate, and combustion use routinely to run large-scale simulations. Unfortunately, while the performance efficiency of GPUs is well understood, their resilience characteristics in a large-scale computing system have not been fully evaluated. We present a detailed study to provide a thorough understanding of GPU errors on a large-scale GPU-enabled system. Our data was collected from the Titan supercomputer at the Oak Ridge Leadership Computing Facility and a GPU cluster at the Los Alamos National Laboratory. We also present results from our extensive neutron-beam tests, conducted at Los Alamos Neutron Science Center (LANSCE) and at ISIS (Rutherford Appleron Laboratories, UK), to measure the resilience of different generations of GPUs. We present several findings from our field data and neutron-beam experiments, and discuss the implications of our results for future GPU architects, current and future HPC computing facilities, and researchers focusing on GPU resilience.

Understanding GPU errors on large-scale HPC systems and the implications for system design and operation

Graphics processing units (GPUs) are playing a critical role in convolutional neural networks (CNNs) for image detection. As GPU-enabled CNNs move into safety-critical environments, reliability is becoming a growing concern. In this paper, we evaluate and propose strategies to improve the reliability of object detection algorithms, as run on three NVIDIA GPU architectures. We consider three algorithms: 1) you only look once; 2) a faster region-based CNN (Faster R-CNN); and 3) a residual network, exposing live hardware to neutron beams. We complement our beam experiments with fault injection to better characterize fault propagation in CNNs. We show that a single fault occurring in a GPU tends to propagate to multiple active threads, significantly reducing the reliability of a CNN. Moreover, relying on error correcting codes dramatically reduces the number of silent data corruptions (SDCs), but does not reduce the number of critical errors (i.e., errors that could potentially impact safety-critical applications). Based on observations on how faults propagate on GPU architectures, we propose effective strategies to improve CNN reliability. We also consider the benefits of using an algorithm-based fault-tolerance technique for matrix multiplication, which can correct more than 87% of the critical SDCs in a CNN, while redesigning maxpool layers of the CNN to detect up to 98% of critical SDCs.

Analyzing and Increasing the Reliability of Convolutional Neural Networks on GPUs

The book is based on a laboratory course in nuclear physics given to advanced students. It treats the experimental techniques and instrumentationmost often used in nuclear and particle physics experiments as well as in various other experimental sciences. It provides most useful results and formulae, technical know-how and informative details on -interactionof radiation in matter; - radion protection and radioactive sources; - statistics for the interpretation and analysis of data; - principles and operation of the main types of detectors (ionization, scintillation and semiconductor detectors); - nuclear electronics instrumentation (NIM, CAMAC); - various systems and techniques for experiments. Thanks to the author's long teaching experience the material is presented in a verypractical, hands-on way making the book a useful text and lab companion for students and experienced scientists.

Detectors for Particle Radiation and Introduction to Experimental Particle Physics and Techniques for Nuclear and Particle Physics Experiments: A How‐To Approach

A survey on modeling and improving reliability of DNN algorithms and accelerators

Graphics processing units (GPUs) are increasingly attractive for both safety-critical and High-Performance Computing applications. GPU reliability is a primary concern for both the automotive and aerospace markets and is becoming an issue also for supercomputers. In fact, the high number of devices in large data centers makes the probability of having at least a device corrupted to be very high. In this paper, we aim at giving novel insights on GPU reliability by evaluating the neutron sensitivity of modern GPUs memory structures, highlighting pattern dependence and multiple errors occurrences. Additionally, a wide set of parallel codes are exposed to controlled neutron beams to measure GPUs operative error rates. From experimental data and algorithm analysis we derive general insights on parallel algorithms and programming approaches reliability. Finally, error-correcting code, algorithm-based fault tolerance, and duplication with comparison hardening strategies are presented and evaluated on GPUs through radiation experiments. We present and compare both the reliability improvement and imposed overhead of the selected hardening solutions.

Evaluation and Mitigation of Radiation-Induced Soft Errors in Graphics Processing Units

Neural networks are becoming an attractive solution for automatizing vehicles in the automotive, military, and aerospace markets. Thanks to their low-cost, low-power consumption, and flexibility, field-programmable gate arrays (FPGAs) are among the promising devices to implement neural networks. Unfortunately, FPGAs are also known to be susceptible to radiation-induced errors. In this paper, we evaluate the effects of radiation-induced errors in the output correctness of two neural networks [Iris Flower artificial neural network (ANN) and Modified National Institute of Standards and Technology (MNIST) convolutional neural network (CNN)] implemented in static random-access memory-based FPGAs. In particular, we notice that radiation can induce errors that modify the output of the network with or without affecting the neural network’s functionality. We call the former critical errors and the latter tolerable errors. Through exhaustive fault injection, we identify the portions of Iris Flower ANN and MNIST CNN implementation on FPGAs that are more likely, once corrupted, to generate a critical or a tolerable error. Based on this analysis, we propose a selective hardening strategy that triplicates only the most vulnerable layers of the neural network. With neutron radiation testing, our selective hardening solution was able to mask 40% of faults with a marginal 8% overhead in one of our tested neural networks.

Selective Hardening for Neural Networks in FPGAs

In this paper we assess the neutron sensitivity of Graphics Processing Units (GPUs) when executing a Fast Fourier Transform (FFT) algorithm, and propose specific software-based hardening strategies to reduce its failure rate. Our research is motivated by experimental results with an unhardened FFT that demonstrate a majority of multiple errors in the output in the case of failures, which are caused by data dependencies. In addition, the use of the built-in error-correction code (ECC) showed a large overhead, and proved to be insufficient to provide high reliability. Experimental results with the hardened algorithm show a two orders of magnitude failure rate improvement over the original algorithm (one order of magnitude over ECC) and an overhead 64% smaller than ECC.

Software-Based Hardening Strategies for Neutron Sensitive FFT Algorithms on GPUs

Charge-collection measurements at the VESUVIO instrument at ISIS are described. Neutron SEU cross sections in SRAM-based FPGAs are measured. Results are compared to equivalent data from Los Alamos Neutron Science Center ICE House. The rate of single-event effects due to fast neutrons at VESUVIO is approximately 15% of that at LANSCE. In addition there is a strong thermal and epithermal component, sufficient to cause many events in devices containing small amounts of 10B. The effects of low-energy neutrons on a commercial CCD contaminated with traces of 10B are described. Cadmium shielding is found to be incompletely effective in separating the effects of fast and slow neutrons, and the implications for testing protocols and instrument design are discussed.

Charge-Collection and Single-Event Upset Measurements at the ISIS Neutron Source

This paper presents and analyzes the results of neutron experiments on 40nm Graphic Processing Units. We have measured the internal memory resources cross sections, and define a new threads cross section to characterize the computing units sensitivity to radiation. We experimentally evaluate the matrix multiplication application error rate and built an analytical model to predict algorithms neutron-induced failures.

https://ieeexplore.ieee.org/iel5/6353295/6353706/06353714.pdf

Neutron-Induced Soft Errors in Graphic Processing Units

We present the results of accelerated radiation testing on an AMD accelerated processing unit, three Nvidia graphic processing units, an Intel accelerator, a field-programmable gate array, and two double-data-rate memories under thermal and high-energy neutrons separately. The sensitivity depends on the device type and the code being executed and we show that thermal neutrons contribute to the error rate of modern computing devices under certain conditions.

C. Frost

Papers

Selective Hardening for Neural Networks in FPGAs

Software-Based Hardening Strategies for Neutron Sensitive FFT Algorithms on GPUs

Charge-Collection and Single-Event Upset Measurements at the ISIS Neutron Source

Neutron-Induced Soft Errors in Graphic Processing Units

High-Energy Versus Thermal Neutron Contribution to Processor and Memory Error Rates