Binary Neural Networks: A Survey

The volume of convolutional neural network (CNN) models proposed for face recognition has been continuously growing larger to better fit large amount of training data. When training data are obtained from internet, the labels are likely to be ambiguous and inaccurate. This paper presents a Light CNN framework to learn a compact embedding on the large-scale face data with massive noisy labels. First, we introduce a variation of maxout activation, called Max-Feature-Map (MFM), into each convolutional layer of CNN. Different from maxout activation that uses many feature maps to linearly approximate an arbitrary convex activation function, MFM does so via a competitive relationship. MFM can not only separate noisy and informative signals but also play the role of feature selection between two feature maps. Second, three networks are carefully designed to obtain better performance meanwhile reducing the number of parameters and computational costs. Lastly, a semantic bootstrapping method is proposed to make the prediction of the networks more consistent with noisy labels. Experimental results show that the proposed framework can utilize large-scale noisy data to learn a Light model that is efficient in computational costs and storage spaces. The learned single network with a 256-D representation achieves state-of-the-art results on various face benchmarks without fine-tuning. The code is released on this https URL.

A Light CNN for Deep Face Representation with Noisy Labels

In this work, we review Binarized Neural Networks (BNNs). BNNs are deep neural networks that use binary values for activations and weights, instead of full precision values. With binary values, BNNs can execute computations using bitwise operations, which reduces execution time. Model sizes of BNNs are much smaller than their full precision counterparts. While the accuracy of a BNN model is generally less than full precision models, BNNs have been closing accuracy gap and are becoming more accurate on larger datasets like ImageNet. BNNs are also good candidates for deep learning implementations on FPGAs and ASICs due to their bitwise efficiency. We give a tutorial of the general BNN methodology and review various contributions, implementations and applications of BNNs.

A Review of Binarized Neural Networks

The increasing demand for on-chip edge intelligence has motivated the exploration of algorithmic techniques and specialized hardware to reduce the computation energy of current machine learning models. In particular, deep spiking neural networks (SNNs) have gained interest because their event-driven hardware implementations can consume very low energy. However, minimizing average spiking activity and thus energy consumption while preserving accuracy in deep SNNs remains a significant challenge and opportunity. This paper proposes a novel two-step SNN compression technique to reduce their spiking activity while maintaining accuracy that involves compressing specifically-designed artificial neural networks (ANNs) that are then converted into the target SNNs. Our approach uses an ultra-high ANN compression technique that is guided by the attention-maps of an uncompressed meta-model. We then evaluate the firing threshold of each ANN layer and start with the trained ANN weights to perform a sparse-learning-based supervised SNN training to minimize the number of time steps required while retaining compression. To evaluate the merits of the proposed approach, we performed experiments with variants of VGG and ResNet, on both CIFAR-10 and CIFAR-100, and VGG16 on Tiny-ImageNet. SNN models generated through the proposed technique yield state-of-the-art compression ratios of up to 33.4× with no significant drop in accuracy compared to baseline unpruned counterparts. As opposed to the existing SNN pruning methods we achieve up to 8.3× better compression with no drop in accuracy. Moreover, compressed SNN models generated by our methods can have up to 12.2× better compute energy-efficiency compared to ANNs that have a similar number of parameters.

/pdf/spike-thrift-towards-energy-efficient-deep-spiking-neural-358n6p1q2z.pdf

Spike-Thrift: Towards Energy-Efficient Deep Spiking Neural Networks by Limiting Spiking Activity via Attention-Guided Compression

In the recent years, the spiking neural network (SNN) has attracted increasing attention due to its low energy consumption and online learning potential. However, the design of SNN processor has not been thoroughly investigated in the past, resulting in limited performance and energy consumption. In this work, a fast and energy-efficient SNN processor with adaptive clock/event-driven computation scheme and online learning capability has been proposed. Several techniques have been proposed to reduce the computation time and energy consumption, including Adaptive Clock- and Event-Driven Computing Scheme, Neighboring PE Borrowing Technique, Compressed Spike Routing Technique and Reconfigurable PE for Inference and Learning. Implemented on a Virtex-7 FPGA, the proposed design achieves computation time of 3.15 ms/image, inference energy consumption of $0.028~\mu $ J/synapse/image and online learning energy consumption of $0.297~\mu $ J/synapse/image for the MNIST 10-class dataset, which outperform several state-of-the-art SNN processors. The proposed SNN processor is suitable for real-time and energy-constrained applications.

A Fast and Energy-Efficient SNN Processor With Adaptive Clock/Event-Driven Computation Scheme and Online Learning

In recent researches, binarized neural network (BNN) has been proposed to address the massive computations and large memory footprint problem of the convolutional neural network (CNN). Several works have designed specific BNN accelerators and showed very promising results. Nevertheless, only part of the neural network is binarized in their architecture and the benefits of binary operations were not fully exploited. In this work, we propose the first fully binarized convolutional neural network accelerator (FBNA) architecture, in which all convolutional operations are binarized and unified, even including the first layer and padding. The fully unified architecture provides more resource, parallelism and scalability optimization opportunities. Compared with the state-of-the-art BNN accelerator, our evaluation results show 3.1x performance, 5.4x resource efficiency and 4.9x power efficiency on CIFAR-10.

FBNA: A Fully Binarized Neural Network Accelerator

As the feature size of the semiconductor process is scaling down to 10nm and below, it is possible to assemble systems with high performance processors that can theoretically provide computational power of up to tens of PLOPS. However, the power consumption of these systems is also rocketing up to tens of millions watts, and the actual performance is only around 60% of the theoretical performance. Today, power efficiency and sustained performance have become the main foci of processor designers. Traditional computing architecture such as superscalar and GPGPU are proven to be power inefficient, and there is a big gap between the actual and peak performance. In this paper, we present the MaPU architecture, a novel architecture which is suitable for data-intensive computing with great power efficiency and sustained computation throughput. To achieve this goal, MaPU attempts to optimize the application from a system perspective, including the hardware, algorithm and corresponding program model. It uses an innovative multi-granularity parallel memory system with intrinsic shuffle ability, cascading pipelines with wide SIMD data paths and a state-machine-based program model. When executing typical signal processing algorithms, a single MaPU core implemented with a 40nm process exhibits a sustained performance of 134 GLOPS while consuming only 2.8 W in power, which increases the actual power efficiency by an order of magnitude comparable with the traditional CPU and GPGPU.

MaPU: A novel mathematical computing architecture

Deep convolutional neural networks (CNNs) have shown state-of-the-art accuracy for various computer vision and speech tasks. However, CNNs are computation-intensive and energy-inefficient which are difficult to be deployed in real-time systems. Event-driven Spiking Neural Networks (SNNs) are extremely power efficient, which provides an alternative for ultra-low power applications. But effective training methods for SNN are still lacking. Due to its spatio-temporal feature of SNN, conventional training method for CNN can not be employed in SNN. To address this problem, some researchers proposed to convert the corresponding weights of trained CNNs into the synapse weights of SNNs (CNNs-SNNs). Nevertheless, limited by the [0, 1] constraints on the SNN neuron outputs, the accuracy of the converted SNNs is impaired. Besides, as the SNN network becomes deeper, the convergence speed of SNN inference are unacceptably slow. In this work, we proposed an innovative deep multi-strength SNN (M-SNN) structure which relaxes the restriction of the neuron output spike strength while the event-driven feature for low-power implementations is maintained. Using this architecture, large scale SNN can be converted from CNN with comparable accuracy and fast inference speed. The evaluation results show 3.7 × convergence speedup. Moreover, with multi-strength spike, aggressive pruning strategies can be applied to reduce the computational operations by almost 85% while maintaining the same accuracy.

Fast and Efficient Deep Sparse Multi-Strength Spiking Neural Networks with Dynamic Pruning

Although the convolutional neural network (CNN) has exhibited outstanding performance in various applications, the deployment of CNN on embedded and mobile devices is limited by the massive computa...

A High-Efficiency FPGA-Based Accelerator for Binarized Neural Network

Deep convolutional neural networks (ConvNets) have achieved the state-of-the-art performance on many real-world applications. However, significant computation and storage demands are required by ConvNets. Spiking neural networks (SNNs), with sparsely activated neurons and event-driven computations, show great potential to take advantage of the ultra- low power spike-based hardware architectures. Yet, training SNN with similar accuracy as ConvNets is difficult. Recent researchers have demonstrated the work of converting ConvNets to SNNs (CNN-SNN conversion) with similar accuracy. However, the energy-efficiency of the converted SNNs is impaired by the increased classification latency. In this paper, we focus on optimizing the classification latency of the converted SNNs. First, we propose a restricted output training method to normalize the converted weights dynamically in the CNN-SNN training phase. Second, false spikes are identified and the false spike inhibition theory is derived to speedup the convergence of the classification process. Third, we propose a temporal max pooling method to approximate the max pooling operation in ConvNets without accuracy loss. The evaluation shows that the converted SNNs converge in about 30 time-steps and achieve the best classification accuracy of 94% on CIFAR -10 dataset.

Hong Ma

Papers

FBNA: A Fully Binarized Neural Network Accelerator

MaPU: A novel mathematical computing architecture

Fast and Efficient Deep Sparse Multi-Strength Spiking Neural Networks with Dynamic Pruning

A High-Efficiency FPGA-Based Accelerator for Binarized Neural Network

Low Latency Spiking ConvNets with Restricted Output Training and False Spike Inhibition