“Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks”の学習報告

From the Publisher: 
The accessible presentation of this book gives both a general view of the entire computer vision enterprise and also offers sufficient detail to be able to build useful applications. Users learn techniques that have proven to be useful by first-hand experience and a wide range of mathematical methods. A CD-ROM with every copy of the text contains source code for programming practice, color images, and illustrative movies. Comprehensive and up-to-date, this book includes essential topics that either reflect practical significance or are of theoretical importance. Topics are discussed in substantial and increasing depth. Application surveys describe numerous important application areas such as image based rendering and digital libraries. Many important algorithms broken down and illustrated in pseudo code. Appropriate for use by engineers as a comprehensive reference to the computer vision enterprise.

Computer vision : a modern approach = 计算机视觉 : 一种现代的方法

비만아 부모의 돌봄 경험: q 방법론

Many architects believe that major improvements in cost-energy-performance must now come from domain-specific hardware. This paper evaluates a custom ASIC---called a Tensor Processing Unit (TPU)---deployed in datacenters since 2015 that accelerates the inference phase of neural networks (NN). The heart of the TPU is a 65,536 8-bit MAC matrix multiply unit that offers a peak throughput of 92 TeraOps/second (TOPS) and a large (28 MiB) software-managed on-chip memory. The TPU's deterministic execution model is a better match to the 99th-percentile response-time requirement of our NN applications than are the time-varying optimizations of CPUs and GPUs (caches, out-of-order execution, multithreading, multiprocessing, prefetching, ...) that help average throughput more than guaranteed latency. The lack of such features helps explain why, despite having myriad MACs and a big memory, the TPU is relatively small and low power. We compare the TPU to a server-class Intel Haswell CPU and an Nvidia K80 GPU, which are contemporaries deployed in the same datacenters. Our workload, written in the high-level TensorFlow framework, uses production NN applications (MLPs, CNNs, and LSTMs) that represent 95% of our datacenters' NN inference demand. Despite low utilization for some applications, the TPU is on average about 15X - 30X faster than its contemporary GPU or CPU, with TOPS/Watt about 30X - 80X higher. Moreover, using the GPU's GDDR5 memory in the TPU would triple achieved TOPS and raise TOPS/Watt to nearly 70X the GPU and 200X the CPU.

In-Datacenter Performance Analysis of a Tensor Processing Unit

Memristive devices are electrical resistance switches that can retain a state of internal resistance based on the history of applied voltage and current. These devices can store and process information, and offer several key performance characteristics that exceed conventional integrated circuit technology. An important class of memristive devices are two-terminal resistance switches based on ionic motion, which are built from a simple conductor/insulator/conductor thin-film stack. These devices were originally conceived in the late 1960s and recent progress has led to fast, low-energy, high-endurance devices that can be scaled down to less than 10 nm and stacked in three dimensions. However, the underlying device mechanisms remain unclear, which is a significant barrier to their widespread application. Here, we review recent progress in the development and understanding of memristive devices. We also examine the performance requirements for computing with memristive devices and detail how the outstanding challenges could be met.

Memristive devices for computing

As Machine Learning (ML) becomes pervasive in the era of artificial intelligence, ML specific tools and frameworks are required for architectural research. This paper introduces NeuroMeter, an integrated power, area, and timing modeling framework for ML accelerators. NeuroMeter models the detailed architecture of ML accelerators and generates a fast and accurate estimation on power, area, and chip timing. Meanwhile, it also enables the runtime analysis of system-level performance and efficiency when the runtime activity factors are provided. NeuroMeter’s micro-architecture model includes fundamental components of ML accelerators, including systolic array based tensor units (TU), reduction trees (RT), and 1D vector units (VU). NeuroMeter has accurate modeling results, with the average power and area estimation errors below 10% and 17% respectively when validated against TPU-v1, TPU-v2, and Eyeriss.Leveraging the NeuroMeter’s new capabilities on architecting manycore ML accelerators, this paper presents the first in-depth study on the design space and tradeoffs of “Brawny and Wimpy” inference accelerators in datacenter scenarios with the insights that are otherwise difficult to discover without NeuroMeter. Our study shows that brawny designs with 64x64 systolic arrays are the most performant and efficient for inference tasks in the 28nm datacenter architectural space with a 500mm2 die area budget. Our study also reveals important tradeoffs between performance and efficiency. For datacenter accelerators with low batch inference, a small $(\sim 16$%) sacrifice of system performance (in achieved Tera OPerations per Second, aka TOPS) can lead to more than a 2x efficiency improvement (in achieved TOPS/TCO). To showcase NeuroMeter’s capability to model a wide range of diverse ML accelerator architectures, we also conduct a followon mini-case study on implications of sparsity on different ML accelerators, demonstrating wimpier accelerator architectures benefit more readily from sparsity processing despite their lower achievable raw energy efficiency.

NeuroMeter: An Integrated Power, Area, and Timing Modeling Framework for Machine Learning Accelerators Industry Track Paper

As technology scales, modern massive parallel processing (MPP) systems are facing large system overhead caused by high failure rates. To provide the system-level fault tolerance, the traditional in-disk checkpointing/restart schemes are usually adopted by periodically dumping system states and memory contents to hard disk drives (HDDs). When errors occur, the system can be restored by reading checkpoints from HDDs. The low bandwidth and slow speed of HDDs are now becoming the major bottleneck for the MPP system performance. Consequently, novel checkpointing schemes are need to facilitate the move from Petascale computing to Exascale computing. We have proposed a 3D memory stacking method [1] that leverage the massive number of TSVs between memory layers to help high-bandwidth checkpointing/restore. To validate the proposed scheme, we design a 2-layer TSV-based SRAM/SRAM 3D-stacked chip to mimic the high-bandwidth and fast data transfer from one memory layer to another memory layer, so that the inmemory checkpointing/restartrestore scheme can be enabled for the future exascale computing. The capacity of each SRAM layer is 1Mbit. Each layer contains 64 banks, with each bank contains 256 words and the word length is 64-bit. The final footprint including I/O pad is 2.9mm×2mm. The SRAM dies were taped out in GlobalFoundries using its 130nm low power process, and the 3D stacking was done by using Tezzaron's TSV technology. The prototyping chip can perform checkpointing/restart at the speed of 4K/cycle with 1Ghz clock.

3D memory stacking for fast checkpointing/restore applications

The impact of process variations on circuit timing increases rapidly as technology scales. Consequently, it is important to consider timing variations at the early stages of circuit designs. Conventional high level synthesis relies on the worst-case delay analysis to guide the design space exploration, however, such worst-case timing analysis can results in overly conservative designs with pessimistic performance estimation. This paper presents a 0-1 integer linear programming (ILP) formulation that aims at reducing the impact of timing variations in high-level synthesis, by integrating overall timing yield constraints into scheduling and resource binding. The proposed approach focuses on how to achieve the maximum performance (minimum latency) under given timing yield constraints with affordable computation time. Experiment results show that significant latency reduction is achieved under different timing yield constraints, compared to traditional worst-case based approach.

ILP-based scheme for timing variation-aware scheduling and resource binding

Exploiting sparsity enables hardware systems to run neural networks faster and more energy-efficiently. However, most prior sparsity-centric optimization techniques only accelerate the forward pass of neural networks and usually require an even longer training process with iterative pruning and retraining. We observe that artificially inducing sparsity in the gradients of the gates in an LSTM cell has little impact on the training quality. Further, we can enforce structured sparsity in the gate gradients to make the LSTM backward pass up to 45% faster than the state-of-the-art dense approach and 168% faster than the state-of-the-art sparsifying method on modern GPUs. Though the structured sparsifying method can impact the accuracy of a model, this performance gap can be eliminated by mixing our sparse training method and the standard dense training method. Experimental results show that the mixed method can achieve comparable results in a shorter time span than using purely dense training.

Structurally Sparsified Backward Propagation for Faster Long Short-Term Memory Training.

Emerging nonvolatile memory (NVM), such as spintransfer torque magnetic Memory (STT-RAM), phase-change memory (PCM), and resistive memory (ReRAM), are widely expected to become the next generation cache and main memory, in order to migrate the “power wall” and overcome the DRAM stability challenge. Previous effort has well explored NVM’s feature of ultra-low leakage and high density at various memory hierarchy. Furthermore, challenges such as asymmetric read/write, expensive write operation, and limited lifetime have also been tackled. However, the benefit from NVM’s nonvolatility has never been fully exploited. This paper points out the potential benefit by leveraging nonvolatility for architecture design. Two case studies are described. The first one is to leveraging multi-level cell (MLC) STT-RAM for ultra-low overhead local checkpointing. The second one is persistent memory design, which support persistency in NVM based main memory. Potential benefit and design challenge for those two cases are described. Future research topic around exploring NVM’s nonvolatility is also discussed.

Yuan Xie

Papers

NeuroMeter: An Integrated Power, Area, and Timing Modeling Framework for Machine Learning Accelerators Industry Track Paper

3D memory stacking for fast checkpointing/restore applications

ILP-based scheme for timing variation-aware scheduling and resource binding

Structurally Sparsified Backward Propagation for Faster Long Short-Term Memory Training.

Leveraging nonvolatility for architecture design with emerging NVM