Pruning and quantization for deep neural network acceleration: A survey

Quantization can improve the execution latency and energy efficiency of neural networks on both commodity GPUs and specialized accelerators. The majority of existing literature focuses on training quantized DNNs, while this work examines the less-studied topic of quantizing a floating-point model without (re)training. DNN weights and activations follow a bell-shaped distribution post-training, while practical hardware uses a linear quantization grid. This leads to challenges in dealing with outliers in the distribution. Prior work has addressed this by clipping the outliers or using specialized hardware. In this work, we propose outlier channel splitting (OCS), which duplicates channels containing outliers, then halves the channel values. The network remains functionally identical, but affected outliers are moved toward the center of the distribution. OCS requires no additional training and works on commodity hardware. Experimental evaluation on ImageNet classification and language modeling shows that OCS can outperform state-of-the-art clipping techniques with only minor overhead.

/pdf/improving-neural-network-quantization-without-retraining-1jgdol3jqf.pdf

Improving Neural Network Quantization without Retraining using Outlier Channel Splitting

Some embodiments of the invention provide a configurable integrated circuit (“IC”). This IC includes several configurable logic circuits for receiving configuration data and configurably performing a set of operations based on the configuration data. It also includes several hybrid circuits. Each particular hybrid circuit has: (1) a set of inputs, (2) a set of outputs for selectively connecting to the set of inputs, and (3) a set of select lines for receiving select signals that direct the hybrid circuit to connect the input set to the output set in a particular manner. At least one select signal is for controllably receiving configuration data and at least one select line is for controllably receiving signals generated by the configurable logic circuits.

Hybrid configurable circuit for a configurable IC

Some embodiments of the invention provide a configurable integrated circuit (“IC”). This IC includes several configurable circuits for receiving configuration data and configurably performing a set of operations based on the configuration data. It also includes several hybrid circuits. Each particular hybrid circuit can interchangeably perform as either a logic circuit or an interconnect circuit in the configurable IC.

Hybrid Logic/Interconnect Circuit in a Configurable IC

Logic design apparatus and method provides serial multiplexer chains in a programmable logic fabric, each element in the chain either selects output of block, or passes output from earlier element of the chain. Select line is a decoder structure or output from configurable function generator that is configured at power-on to create correct selection. Using such structure, larger multiplexer, including priority multiplexers, tristate buses or larger look-up tables (LUTs) can be created. These novel structures can implement priority, non-priority or tristate multiplexers.

Versatile multiplexer-structures in programmable logic using serial chaining and novel selection schemes

A programmable logic structure is disclosed that has a set of dedicated lines which extend internally throughout different dedicated logic cells within a logic and routing block (LRB), extend from a previous logic routing block to the present logic and routing block, or extend from the present logic and routing block to the next logic and routing block. One set of dedicated lines from a first logic and routing block can be stitched to another set of dedicated lines of a second logic and routing block for extending the reach as well as bypassing a logic and routing block, or bypassing a dedicated logic cell in the same logic and routing block. The dedicated lines between logic and routing blocks allow a logic and routing block to receive more inputs from its own switch box or to drive more outputs than provided by the logic and routing block as specified by a given function.

Programmable logic and routing blocks with dedicated lines

A dedicated logic cell in a programmable logic structure is described that comprises the following primary components: a configurable logic function or look-up table (LL), a dedicated logic function (DL), a sequential logic function (LS), and a control logic function (LC). In this illustration, the dedicated logic cell comprises two configurable logic functions, two sequential logic functions, a dedicate logic function, and a control logic function. In a first embodiment, the dedicated logic cell is constructed with a combination of configurable logic functions that are coupled to a dedicated logic function in order to perform a four 2-input function, an AND function, an OR function, or an XOR function. In a second embodiment, the dedicated logic cell is constructed with a combination of configurable logic functions that are coupled to a dedicated logic function in order to perform a four 2-to-1 multiplexer function. In a third embodiment, the dedicated logic cell is constructed with a plurality of configurable logic functions that operate as a two 6-input function with separate inputs. In a fourth embodiment, the dedicated logic cell is constructed with a combination of a configurable logic function with sequential logic functions that operate as a loadable, resettable, clearable shift register. In a fifth embodiment, the dedicated logic cell is constructed with a combination of configurable logic functions, a dedicated logic function, and sequential logic functions that operate as an accumulator.

Dedicated logic cells employing configurable logic and dedicated logic functions

Deep learning as a means to inferencing has proliferated thanks to its versatility and ability to approach or exceed human-level accuracy. These computational models have seemingly insatiable appetites for computational resources not only while training, but also when deployed at scales ranging from data centers all the way down to embedded devices. As such, increasing consideration is being made to maximize the computational efficiency given limited hardware and energy resources and, as a result, inferencing with reduced precision has emerged as a viable alternative to the IEEE 754 Standard for Floating-Point Arithmetic. We propose a quantization scheme that allows inferencing to be carried out using arithmetic that is fundamentally more efficient when compared to even half-precision floating-point. Our quantization procedure is significant in that we determine our quantization scheme parameters by calibrating against its reference floating-point model using a single inference batch rather than (re)training and achieve end-to-end post quantization accuracies comparable to the reference model.

Quantizing Convolutional Neural Networks for Low-Power High-Throughput Inference Engines

In this paper, we will describe the architectural, software, performance, and implementation challenges and solutions and current research on the use of programmable logic to enable deep learning applications. First a discussion of characteristics of building a deep learning system will described. Next architectural choices will be explained for how a FPGA fabric can efficiently solve deep learning tasks. Finally specific techniques for how DSPs, memories and are used in high performance applications will be described.

Elliott Delaye

Papers

Versatile multiplexer-structures in programmable logic using serial chaining and novel selection schemes

Programmable logic and routing blocks with dedicated lines

Dedicated logic cells employing configurable logic and dedicated logic functions

Quantizing Convolutional Neural Networks for Low-Power High-Throughput Inference Engines

Deep learning challenges and solutions with xilinx FPGAs