A number of recent efforts have attempted to design accelerators for popular machine learning algorithms, such as those involving convolutional and deep neural networks (CNNs and DNNs). These algorithms typically involve a large number of multiply-accumulate (dot-product) operations. A recent project, DaDianNao, adopts a near data processing approach, where a specialized neural functional unit performs all the digital arithmetic operations and receives input weights from adjacent eDRAM banks.This work explores an in-situ processing approach, where memristor crossbar arrays not only store input weights, but are also used to perform dot-product operations in an analog manner. While the use of crossbar memory as an analog dot-product engine is well known, no prior work has designed or characterized a full-fledged accelerator based on crossbars. In particular, our work makes the following contributions: (i) We design a pipelined architecture, with some crossbars dedicated for each neural network layer, and eDRAM buffers that aggregate data between pipeline stages. (ii) We define new data encoding techniques that are amenable to analog computations and that can reduce the high overheads of analog-to-digital conversion (ADC). (iii) We define the many supporting digital components required in an analog CNN accelerator and carry out a design space exploration to identify the best balance of memristor storage/compute, ADCs, and eDRAM storage on a chip. On a suite of CNN and DNN workloads, the proposed ISAAC architecture yields improvements of 14.8×, 5.5×, and 7.5× in throughput, energy, and computational density (respectively), relative to the state-of-the-art DaDianNao architecture.

ISAAC: a convolutional neural network accelerator with in-situ analog arithmetic in crossbars

Capture-Recapture and Removal Methods for Sampling Closed Populations

This paper describes an ultra-low power SAR ADC for medical implant devices. To achieve the nano-watt range power consumption, an ultra-low power design strategy has been utilized, imposing maximum simplicity on the ADC architecture, low transistor count and matched capacitive DAC with a switching scheme which results in full-range sampling without switch bootstrapping and extra reset voltage. Furthermore, a dual-supply voltage scheme allows the SAR logic to operate at 0.4 V, reducing the overall power consumption of the ADC by 15% without any loss in performance. The ADC was fabricated in 0.13-μm CMOS. In dual-supply mode (1.0 V for analog and 0.4 V for digital), the ADC consumes 53 nW at a sampling rate of 1 kS/s and achieves the ENOB of 9.1 bits. The leakage power constitutes 25% of the 53-nW total power.

/pdf/a-53-nw-9-1-enob-1-ks-s-sar-adc-in-0-13-mu-m-cmos-for-49q2yef4do.pdf

A 53-nW 9.1-ENOB 1-kS/s SAR ADC in 0.13- $\mu$ m CMOS for Medical Implant Devices

Exploiting model sparsity to reduce ineffectual computation is a commonly used approach to achieve energy efficiency for DNN inference accelerators. However, due to the tightly coupled crossbar structure, exploiting sparsity for ReRAM-based NN accelerator is a less explored area. Existing architectural studies on ReRAM-based NN accelerators assume that an entire crossbar array can be activated in a single cycle. However, due to inference accuracy considerations, matrix-vector computation must be conducted in a smaller granularity in practice, called Operation Unit (OU). An OU-based architecture creates a new opportunity to exploit DNN sparsity. In this paper, we propose the first practical Sparse ReRAM Engine that exploits both weight and activation sparsity. Our evaluation shows that the proposed method is effective in eliminating ineffectual computation, and delivers significant performance improvement and energy savings.

Sparse ReRAM engine: joint exploration of activation and weight sparsity in compressed neural networks

Analog hardware accelerators, which perform computation within a dense memory array, have the potential to overcome the major bottlenecks faced by digital hardware for data-heavy workloads such as deep learning. Exploiting the intrinsic computational advantages of memory arrays, however, has proven to be challenging principally due to the overhead imposed by the peripheral circuitry and due to the non-ideal properties of memory devices that play the role of the synapse. We review the existing implementations of these accelerators for deep supervised learning, organizing our discussion around the different levels of the accelerator design hierarchy, with an emphasis on circuits and architecture. We explore and consolidate the various approaches that have been proposed to address the critical challenges faced by analog accelerators, for both neural network inference and training, and highlight the key design trade-offs underlying these techniques.

Analog architectures for neural network acceleration based on non-volatile memory

Successive-approximation analog-to-digital converters (SA-ADCs) are widely used in ultra-low-power applications. In this paper, the power consumption and the linearity of capacitive-array digital-to-analog converters (DACs) employed in SA-ADCs are analyzed. Specifically, closed-form formulas for the power consumption as well as the standard deviation of INL and DNL for three commonly-used radix-2 architectures including the effect of parasitic capacitances are presented and the structures are compared. The proposed analysis can be employed in choosing the best architecture and optimizing it in both hand calculations and computer-aided-design tools. Measurement results of previously published works as well as simulation results of a 10-bit 10 kS/s SA-ADC confirm the accuracy of the proposed equations. It will be shown that, in spite of what commonly is assumed, although the total capacitance and the power consumption of those architectures employing attenuating capacitors seem to be smaller than conventional binary-weighted structures, the linearity requirements impose much larger unit capacitance to the structure such that the entire power consumption is larger.

/pdf/analysis-of-power-consumption-and-linearity-in-capacitive-2jq18abhrs.pdf

Analysis of Power Consumption and Linearity in Capacitive Digital-to-Analog Converters Used in Successive Approximation ADCs

This brief presents a power-efficient voltage level-shifter architecture that is capable of converting extremely low levels of input voltages to higher levels. In order to avoid the static power dissipation, the proposed structure uses a current generator that turns on only during the transition times, in which the logic level of the input signal is not corresponding to the output logic level. Moreover, the strength of the pull-up device is decreased when the pull-down device is pulling down the output node in order for the circuit to be functional even for the input voltage lower than the threshold voltage of a MOSFET. The operation of the proposed structure is also analytically investigated. Post-layout simulation results of the proposed structure in a 0.18-μm CMOS technology show that at the input low supply voltage of 0.4 V and the high supply voltage of 1.8 V, the level shifter has a propagation delay of 30 ns, a static power dissipation of 130 pW, and an energy per transition of 327 fJ for a 1-MHz input signal.

/pdf/a-low-power-subthreshold-to-above-threshold-voltage-level-49apwulgtc.pdf

A Low-Power Subthreshold to Above-Threshold Voltage Level Shifter

This letter proposes a new level-shifting structure that can convert extremely low levels of input voltages to high output voltages while maintaining excellent delay and power dissipation. In order to reduce contention and voltage swing in the internal nodes, the proposed circuit uses a diode-connected level shifter between gate terminals of the output inverter. Using a control circuit, only during the high-to-low transitions of the output, a current is forced into the diode-connected device. Measurement results demonstrate that the proposed circuit can consume as small energy as 4.2 fJ/transition with ${V} _{\text {DDL}}$ and ${V} _{\text {DDH}}$ of 0.35 V and 1.1 V, respectively, when implemented in a 40-nm CMOS technology. Furthermore, when fabricated in a 180-nm technology, the level-shifting circuit can convert ${V} _{\text {DDL}}$ as small as 80 mV to 1.8 V without using low-threshold devices.

/pdf/energy-efficient-wide-range-voltage-level-shifters-reaching-52e8mwdy6d.pdf

Energy-Efficient Wide-Range Voltage Level Shifters Reaching 4.2 fJ/Transition

This brief presents a fast and power-efficient voltage level-shifting circuit capable of converting extremely low levels of input voltages into high output voltage levels. The efficiency of the proposed circuit is due to the fact that not only the strength of the pull-up device is significantly reduced when the pull-down device is pulling down the output node, but the strength of the pull-down device is also increased using a low-power auxiliary circuit. Postlayout simulation results of the proposed circuit in a 0.18- $\mu \text{m}$ technology demonstrate a total energy per transition of 157 fJ, a static power dissipation of 0.3 nW, and a propagation delay of 30 ns for input frequency of 1 MHz, low supply voltage level of $V_{\mathrm {{DDL}}}= 0.4$ V, and high supply voltage level of $V_{\mathrm {{DDH}}}= 1.8$ V.

A High-Speed and Power-Efficient Voltage Level Shifter for Dual-Supply Applications

In this paper, the structure of a binary-weighted capacitive digital-to-analog converter (DAC) in a successive-approximation analog-to-digital converter (SA-ADC) is modified to a unary or segmented configuration to reduce the power consumption and improve the static linearity performance. In order to be able to choose the optimum value of the segmentation degree (i.e., the number of unary bits), the power consumption and the static linearity behavior of the segmented architecture as functions of the segmentation degree are analyzed. Circuit-level simulation results are presented to show the accuracy of the proposed equations. It is shown that for moderate and high-resolution ADCs, a segmentation degree of four or five bits is the optimum choice from the power-consumption viewpoint. Simulation results of a 1 V, 10-bit, 100-kS/s SA-ADC show that the power consumption of the entire capacitive DAC and the digital circuit of the segmented implementation with a segmentation degree of 4 is 30% less than the conventional design while the standard deviation of the differential nonlinearity is reduced by a factor of 2√(2).

Mehdi Saberi

Papers

Analysis of Power Consumption and Linearity in Capacitive Digital-to-Analog Converters Used in Successive Approximation ADCs

A Low-Power Subthreshold to Above-Threshold Voltage Level Shifter

Energy-Efficient Wide-Range Voltage Level Shifters Reaching 4.2 fJ/Transition

A High-Speed and Power-Efficient Voltage Level Shifter for Dual-Supply Applications

Segmented Architecture for Successive Approximation Analog-to-Digital Converters