Deep neural networks (DNNs) are currently widely used for many artificial intelligence (AI) applications including computer vision, speech recognition, and robotics. While DNNs deliver state-of-the-art accuracy on many AI tasks, it comes at the cost of high computational complexity. Accordingly, techniques that enable efficient processing of DNNs to improve energy efficiency and throughput without sacrificing application accuracy or increasing hardware cost are critical to the wide deployment of DNNs in AI systems. This article aims to provide a comprehensive tutorial and survey about the recent advances toward the goal of enabling efficient processing of DNNs. Specifically, it will provide an overview of DNNs, discuss various hardware platforms and architectures that support DNNs, and highlight key trends in reducing the computation cost of DNNs either solely via hardware design changes or via joint hardware design and DNN algorithm changes. It will also summarize various development resources that enable researchers and practitioners to quickly get started in this field, and highlight important benchmarking metrics and design considerations that should be used for evaluating the rapidly growing number of DNN hardware designs, optionally including algorithmic codesigns, being proposed in academia and industry. The reader will take away the following concepts from this article: understand the key design considerations for DNNs; be able to evaluate different DNN hardware implementations with benchmarks and comparison metrics; understand the tradeoffs between various hardware architectures and platforms; be able to evaluate the utility of various DNN design techniques for efficient processing; and understand recent implementation trends and opportunities.

Efficient Processing of Deep Neural Networks: A Tutorial and Survey

Processing-in-memory (PIM) is a promising solution to address the "memory wall" challenges for future computer systems. Prior proposed PIM architectures put additional computation logic in or near memory. The emerging metal-oxide resistive random access memory (ReRAM) has showed its potential to be used for main memory. Moreover, with its crossbar array structure, ReRAM can perform matrix-vector multiplication efficiently, and has been widely studied to accelerate neural network (NN) applications. In this work, we propose a novel PIM architecture, called PRIME, to accelerate NN applications in ReRAM based main memory. In PRIME, a portion of ReRAM crossbar arrays can be configured as accelerators for NN applications or as normal memory for a larger memory space. We provide microarchitecture and circuit designs to enable the morphable functions with an insignificant area overhead. We also design a software/hardware interface for software developers to implement various NNs on PRIME. Benefiting from both the PIM architecture and the efficiency of using ReRAM for NN computation, PRIME distinguishes itself from prior work on NN acceleration, with significant performance improvement and energy saving. Our experimental results show that, compared with a state-of-the-art neural processing unit design, PRIME improves the performance by ~2360× and the energy consumption by ~895×, across the evaluated machine learning benchmarks.

PRIME: a novel processing-in-memory architecture for neural network computation in ReRAM-based main memory

The explosion of digital data and the ever-growing need for fast data analysis have made in-memory big-data processing in computer systems increasingly important. In particular, large-scale graph processing is gaining attention due to its broad applicability from social science to machine learning. However, scalable hardware design that can efficiently process large graphs in main memory is still an open problem. Ideally, cost-effective and scalable graph processing systems can be realized by building a system whose performance increases proportionally with the sizes of graphs that can be stored in the system, which is extremely challenging in conventional systems due to severe memory bandwidth limitations. In this work, we argue that the conventional concept of processing-in-memory (PIM) can be a viable solution to achieve such an objective. The key modern enabler for PIM is the recent advancement of the 3D integration technology that facilitates stacking logic and memory dies in a single package, which was not available when the PIM concept was originally examined. In order to take advantage of such a new technology to enable memory-capacity-proportional performance, we design a programmable PIM accelerator for large-scale graph processing called Tesseract. Tesseract is composed of (1) a new hardware architecture that fully utilizes the available memory bandwidth, (2) an efficient method of communication between different memory partitions, and (3) a programming interface that reflects and exploits the unique hardware design. It also includes two hardware prefetchers specialized for memory access patterns of graph processing, which operate based on the hints provided by our programming model. Our comprehensive evaluations using five state-of-the-art graph processing workloads with large real-world graphs show that the proposed architecture improves average system performance by a factor of ten and achieves 87% average energy reduction over conventional systems.

/pdf/a-scalable-processing-in-memory-accelerator-for-parallel-23yasy5934.pdf

A scalable processing-in-memory accelerator for parallel graph processing

Deep neural networks (DNNs) are currently widely used for many artificial intelligence (AI) applications including computer vision, speech recognition, and robotics. While DNNs deliver state-of-the-art accuracy on many AI tasks, it comes at the cost of high computational complexity. Accordingly, techniques that enable efficient processing of DNNs to improve energy efficiency and throughput without sacrificing application accuracy or increasing hardware cost are critical to the wide deployment of DNNs in AI systems. 
This article aims to provide a comprehensive tutorial and survey about the recent advances towards the goal of enabling efficient processing of DNNs. Specifically, it will provide an overview of DNNs, discuss various hardware platforms and architectures that support DNNs, and highlight key trends in reducing the computation cost of DNNs either solely via hardware design changes or via joint hardware design and DNN algorithm changes. It will also summarize various development resources that enable researchers and practitioners to quickly get started in this field, and highlight important benchmarking metrics and design considerations that should be used for evaluating the rapidly growing number of DNN hardware designs, optionally including algorithmic co-designs, being proposed in academia and industry. 
The reader will take away the following concepts from this article: understand the key design considerations for DNNs; be able to evaluate different DNN hardware implementations with benchmarks and comparison metrics; understand the trade-offs between various hardware architectures and platforms; be able to evaluate the utility of various DNN design techniques for efficient processing; and understand recent implementation trends and opportunities.

The high accuracy of deep neural networks (NNs) has led to the development of NN accelerators that improve performance by two orders of magnitude. However, scaling these accelerators for higher performance with increasingly larger NNs exacerbates the cost and energy overheads of their memory systems, including the on-chip SRAM buffers and the off-chip DRAM channels.This paper presents the hardware architecture and software scheduling and partitioning techniques for TETRIS, a scalable NN accelerator using 3D memory. First, we show that the high throughput and low energy characteristics of 3D memory allow us to rebalance the NN accelerator design, using more area for processing elements and less area for SRAM buffers. Second, we move portions of the NN computations close to the DRAM banks to decrease bandwidth pressure and increase performance and energy efficiency. Third, we show that despite the use of small SRAM buffers, the presence of 3D memory simplifies dataflow scheduling for NN computations. We present an analytical scheduling scheme that matches the efficiency of schedules derived through exhaustive search. Finally, we develop a hybrid partitioning scheme that parallelizes the NN computations over multiple accelerators. Overall, we show that TETRIS improves mthe performance by 4.1x and reduces the energy by 1.5x over NN accelerators with conventional, low-power DRAM memory systems.

https://dl.acm.org/doi/pdf/10.1145/3037697.3037702

TETRIS: Scalable and Efficient Neural Network Acceleration with 3D Memory

Multi-core processor performance is limited by memory system bandwidth. The Hybrid Memory Cube is a three-dimensional DRAM architecture that improves latency, bandwidth, power and density. Through-silicon vias (TSVs), 3D packaging and advanced CMOS performance enable a new approach to memory system architecture. Heterogeneous die are stacked with significantly more connections, thereby reducing the distance signals travel.

Hybrid memory cube new DRAM architecture increases density and performance

An interface circuit, which can form part of a memory device or a memory controller, includes a read circuit, a write circuit, and a clocking circuit. The read circuit includes two registers or latches that receive alternate data read from burst EDO or synchronous memory. A multiplexer and read output register provide the data to a CPU or other application. If the memory is burst EDO, then the clocking circuit receives the system clock signal and generates a CAS signal based on positive or negative going edges of the clock signal, depending upon delays inherent in the system in which the present invention is employed. The CAS signal is then used to drive the two read latches. If the memory is synchronous memory, then the clock circuit includes an inverter that inverts the clock signal, and provides the inverted clock signal to the synchronous memory. The inverted clock signal is delayed and then used to drive the two read registers. A write register in the write circuit is driven based on the system clock, and therefore operates separately from the read circuit.

Interface for high speed memory

Systems and methods disclosed herein substantially concurrently transfer a plurality of streams of commands, addresses, and/or data across a corresponding plurality of serialized communication link interfaces (SCLIs) between one or more originating devices or destination devices such as a processor and a switch. At the switch, one or more commands, addresses, or data corresponding to each stream can be transferred to a corresponding destination memory vault controller (MVC) associated with a corresponding memory vault. The destination MVC can perform write operations, read operations, and/or memory vault housekeeping operations independently from concurrent operations associated with other MVCs coupled to a corresponding plurality of memory vaults.

Multi-serial interface stacked-die memory architecture

Memory devices and methods are described that include a stack of memory dies and a logic die. Method and devices described include those that provide for repartitioning the stack of memory dies and storing the new partitions in a memory map. Repartitioning in selected configurations allows portions of memory to be removed from use without affecting the rest of the memory device. Additional devices, systems, and methods are disclosed.

Memory devices and methods for managing error regions

Memory devices and methods are described such as those that monitor and adjust characteristics for various different portions of a given memory device. Examples of different portions include tiles, or arrays, or dies. One memory device and method described includes monitoring and adjusting characteristics of different portions of a 3D stack of memory dies. One characteristic that can be adjusted at multiple selected portions includes refresh rate.

Joe M. Jeddeloh

Papers

Hybrid memory cube new DRAM architecture increases density and performance

Interface for high speed memory

Multi-serial interface stacked-die memory architecture

Memory devices and methods for managing error regions

Variable memory refresh devices and methods