Deeply embedded applications require low-power, low-cost hardware that fits within stringent area constraints. Deep learning has many potential uses in these domains, but introduces significant inefficiencies stemming from off-chip DRAM accesses of model weights. Ideally, models would fit entirely on-chip. However, even with compression, memory requirements for state-of-the-art models make on-chip inference impractical. Due to increased density, emerging eNVMs are one promising solution. We present MaxNVM, a principled co-design of sparse encodings, protective logic, and fault-prone MLC eNVM technologies (i.e., RRAM and CTT) to enable highly-efficient DNN inference. We find bit reduction techniques (e.g., clustering and sparse compression) increase weight vulnerability to faults. This limits the capabilities of MLC eNVM. To circumvent this limitation, we improve storage density (i.e., bits-per-cell) with minimal overhead using protective logic. Tradeoffs between density and reliability result in a rich design space. We show that by balancing these techniques, the weights of large networks are able to reasonably fit on-chip. Compared to a naive, single-level-cell eNVM solution, our highly-optimized MLC memory systems reduce weight area by up to 29×. We compare our technique against NVDLA, a state-of-the-art industry-grade CNN accelerator, and demonstrate up to 3.2× reduced power and up to 3.5× reduced energy per ResNet50 inference.

MaxNVM: Maximizing DNN Storage Density and Inference Efficiency with Sparse Encoding and Error Mitigation

14.1 A 2.9TOPS/W deep convolutional neural network SoC in FD-SOI 28nm for intelligent embedded systems

One of the biggest performance bottlenecks of today’s neural network (NN) accelerators is off-chip memory accesses [11]. In this paper, we propose a method to use multi-level, embedded nonvolatile memory (eNVM) to eliminate all off-chip weight accesses. The use of multi-level memory cells increases the probability of faults. Therefore, we co-design the weights and memories such that their properties complement each other and the faults result in no noticeable NN accuracy loss. In the extreme case, the weights in fully connected layers can be stored using a single transistor. With weight pruning and clustering, we show our technique reduces the memory area by over an order of magnitude compared to an SRAM baseline. In the case of VGG16 (130M weights), we are able to store all the weights in 4.9 mm2, well within the area allocated to SRAM in modern NN accelerators [6].

https://dl.acm.org/doi/pdf/10.1145/3195970.3196083

On-chip deep neural network storage with multi-level eNVM

Machine learning is widely used inmanymodern artificial intelligence applications. Various hardware platforms are implemented to support such applications. Among them, graphics processing unit (GPU) is the most widely used one due to its fast computation speed and compatibility with various algorithms. Field programmable gate arrays (FPGA) show better energy efficiency compared with GPU when computing machine learning algorithm at the cost of low speed. Finally, various application specific integrated circuit (ASIC) architecture is proposed to achieve the best energy efficiency at the cost of less reconfigurability which makes it suitable for special kinds of machine learning algorithms such as a deep convolutional neural network. Finally, analog computing shows a promising methodology to compute large-sized machine learning algorithm due to its low design cost and fast computing speed; however, due to the requirement of the analog-to-digital converter (ADC) in the analog computing, this kind of technique is only applicable to low computation resolution, making it unsuitable for most artificial intelligence (AI) applications.

/pdf/hardware-accelerator-design-for-machine-learning-35kb0r3dnh.pdf

Hardware Accelerator Design for Machine Learning

Configurable input blocks and output blocks and physical layouts are disclosed for analog neural memory systems that utilize non-volatile memory cells. An input block can be configured to support different numbers of arrays arranged in a horizontal direction, and an output block can be configured to support different numbers of arrays arranged in a vertical direction. Adjustable components are disclosed for use in the configurable input blocks and output blocks. Systems and methods are utilized for compensating for leakage and offset in the input blocks and output blocks the in analog neural memory systems.

Output array neuron conversion and calibration for analog neural memory in deep learning artificial neural network

A neural network computing engine having an array of charge-trap-transistor (CTT) elements which are utilized as analog multipliers with all weight values preprogrammed into each CTT element as a CTT threshold voltage, with multiplicator values received from the neural network inference mode. The CTT elements perform computations of a fully connected (FC) neural network with each CTT element representing a neuron. Row resistors for each row of CTT element sum output currents as partial summation results. Counted pulse generators write weight values under control of a pulse generator controller. A sequential analog fabric (SAF) feeds multiple drain voltages in parallel to the CTT array to enable parallel analog computations of neurons. Partial summation results are read by an analog-to-digital converter (ADC).

Memristive neural network computing engine using cmos-compatible charge-trap-transistor (ctt)

A system and method for computing a sparse neural network having a plurality of output layers, each of which has a neuron value. Processing engines (PEs) each have a local memory for storing neurons for use with different weight values in a following cycle. A multiplexer selects between the input neuron or the output of the memory. Output from the multiplexor is received along with a weight input to a multiplier whose output is directed to an integrator. A decomposition technique performs a network computation through the use of intermediate neurons when the input neuron is larger than the local memory capacity, and provides data reuse by reusing neurons stored in local memory. Neural systems can be implemented using a neural index to address each of multiple PEs and a parallel-serial first-in-first-out (FIFO) to serially store values in main memory.

Chang Mau-Chung Frank

Papers

Memristive neural network computing engine using cmos-compatible charge-trap-transistor (ctt)

Architecture to compute sparse neural network