Machine learning (ML) models are widely used in many domains including media processing and generation, computer vision, medical diagnosis, embedded systems, high-performance and scientific computing, and recommendation systems. For efficiently processing these computational- and memory-intensive applications, tensors of these over-parameterized models are compressed by leveraging sparsity, size reduction, and quantization of tensors. Unstructured sparsity and tensors with varying dimensions yield irregular-shaped computation, communication, and memory access patterns; processing them on hardware accelerators in a conventional manner does not inherently leverage acceleration opportunities. This paper provides a comprehensive survey on how to efficiently execute sparse and irregular tensor computations of ML models on hardware accelerators. In particular, it discusses additional enhancement modules in architecture design and software support; categorizes different hardware designs and acceleration techniques and analyzes them in terms of hardware and execution costs; highlights further opportunities in terms of hardware/software/algorithm co-design optimizations and joint optimizations among described hardware and software enhancement modules. The takeaways from this paper include: understanding the key challenges in accelerating sparse, irregular-shaped, and quantized tensors; understanding enhancements in acceleration systems for supporting their efficient computations; analyzing trade-offs in opting for a specific type of design enhancement; understanding how to map and compile models with sparse tensors on the accelerators; understanding recent design trends for efficient accelerations and further opportunities.

Hardware Acceleration of Sparse and Irregular Tensor Computations of ML Models: A Survey and Insights

Machine learning has enabled significant benefits in diverse fields, but, with a few exceptions, has had limited impact on computer architecture. Recent work, however, has explored broader applicability for design, optimization, and simulation. Notably, machine learning based strategies often surpass prior state-of-the-art analytical, heuristic, and human-expert approaches. This paper reviews machine learning applied system-wide to simulation and run-time optimization, and in many individual components, including memory systems, branch predictors, networks-on-chip, and GPUs. The paper further analyzes current practice to highlight useful design strategies and identify areas for future work, based on optimized implementation strategies, opportune extensions to existing work, and ambitious long term possibilities. Taken together, these strategies and techniques present a promising future for increasingly automated architectural design.

A Survey of Machine Learning Applied to Computer Architecture Design.

Data movement between the CPU and main memory is a first-order obstacle against improving performance, scalability, and energy efficiency in modern systems. Computer systems employ a range of techniques to reduce overheads tied to data movement, spanning from traditional mechanisms (e.g., deep multi-level cache hierarchies, aggressive hardware prefetchers) to emerging techniques such as Near-Data Processing (NDP), where some computation is moved close to memory. Our goal is to methodically identify potential sources of data movement over a broad set of applications and to comprehensively compare traditional compute-centric data movement mitigation techniques to more memory-centric techniques, thereby developing a rigorous understanding of the best techniques to mitigate each source of data movement. 
With this goal in mind, we perform the first large-scale characterization of a wide variety of applications, across a wide range of application domains, to identify fundamental program properties that lead to data movement to/from main memory. We develop the first systematic methodology to classify applications based on the sources contributing to data movement bottlenecks. From our large-scale characterization of 77K functions across 345 applications, we select 144 functions to form the first open-source benchmark suite (DAMOV) for main memory data movement studies. We select a diverse range of functions that (1) represent different types of data movement bottlenecks, and (2) come from a wide range of application domains. Using NDP as a case study, we identify new insights about the different data movement bottlenecks and use these insights to determine the most suitable data movement mitigation mechanism for a particular application. We open-source DAMOV and the complete source code for our new characterization methodology at this https URL.

DAMOV: A New Methodology and Benchmark Suite for Evaluating Data Movement Bottlenecks

Decision trees are machine learning models commonly used in various application scenarios. In the era of big data, traditional decision tree induction algorithms are not suitable for learning large-scale datasets due to their stringent data storage requirement. Online decision tree learning algorithms have been devised to tackle this problem by concurrently training with incoming samples and providing inference results. However, even the most up-to-date online tree learning algorithms still suffer from either high memory usage or high computational intensity with dependency and long latency, making them challenging to implement in hardware. To overcome these difficulties, we introduce a new quantile-based algorithm to improve the induction of the Hoeffding tree, one of the state-of-the-art online learning models. The proposed algorithm is lightweight in terms of both memory and computational demand, while still maintaining high generalization ability. A series of optimization techniques dedicated to the proposed algorithm have been investigated from the hardware perspective, including coarse-grained and fine-grained parallelism, dynamic and memory-based resource sharing, pipelining with data forwarding. Following this, we present Hard-ODT, a high-performance, hardware-efficient and scalable online decision tree learning system on a field-programmable gate array (FPGA) with system-level optimization techniques. Performance and resource utilization are modeled for the complete learning system for early and fast analysis of the tradeoff between various design metrics. Finally, we propose a design flow in which the proposed learning system is applied to FPGA run-time power monitoring as a case study. Experimental results show that our proposed algorithm outperforms the state-of-the-art Hoeffding tree learning method, leading to 0.05% to 12.3% improvement in inference accuracy. Real implementation of the complete learning system on the FPGA demonstrates a $384\times $ to $1581\times $ speedup in execution time over the state-of-the-art design. The power modeling strategy with Hard-ODT achieves an average power prediction error within 4.93% of a commercial gate-level power estimation tool.

Hard-ODT: Hardware-Friendly Online Decision Tree Learning Algorithm and System

Field-Programmable Gate Arrays (FPGAs) have enabled the acceleration of important applications in the networking, cloud, and artificial intelligence domains, while providing a flexible fabric that can be reprogrammed on demand. Still, the high static power dissipation of FPGAs driven by Static Random Access Memories (SRAMs) leads them to energy consumption levels that may be unacceptable for several application domains. Reconfigurable fabrics with emerging Resistive RAM (RRAM) technologies have been considered as one of the most promising solutions to address these energy issues of current FPGAs. However, the low endurance and the high variability of these emerging devices present a threat to the demands for reconfiguration cycles of current applications, pushing for novel architectures and design strategies techniques for improving the device's lifetime. To address these challenges, we propose a novel reconfigurable architecture targeting classes of applications that require high flexibility in the field. More specifically, we introduce a reconfigurable architecture based on Ternary Content-Addressable Memories (TCAMs) that meets a double mission: to accelerate and tolerate endurance and variation issues supported by a CAD tool that foresees the reuse of data configuration, allowing for an increased endurance in the field. We present the potential of the proposed architecture and its synthesis flow for processing Regular Expression Matching (REM), widely used in network intrusion detection systems. The results show that the performance can achieve up to 32Gbps throughput at 0.89W, while improving the device's lifetime by two orders of magnitude.

Endurance-Aware RRAM-Based Reconfigurable Architecture using TCAM Arrays

Approximate computing has recently reemerged as a design solution for additional performance and energy improvements at the cost of output quality. In this paper, we propose using a tree-based classification algorithm as an approximation tool for general-purpose applications. We show that, without any hardware support, completely implemented in software, our approach can improve performance by up to 4x (1.95x on average) and reduce EDP by up to 19x (4.04 on average) when compared to precise executions. Besides that, in some cases, our software-based mechanism can even outperform traditional hardware-based Neural Network's state-of-the-art designs.

Employing classification-based algorithms for general-purpose approximate computing

In the past years, several works have proposed custom hardware and software-based techniques for the acceleration of Convolutional Neural Networks (CNNs). Most of these works focus on saving computations by changing the used precision or modifying frame processing. To reach a more aggressive energy reduction, in this paper we propose software-only modifications to the CNNs inference process.Our approach exploits the inherent locality in videos by replacing entire frame computations with a movement prediction algorithm. Furthermore, when a frame must be processed, we avoid energy-demanding floating-point operations, and at the same time reduce memory accesses by employing look-up tables in place of the original convolutions.Using the proposed approach, one can reach significant energy gains of more than 25× for security cameras, and 12× for moving vehicles applications, with only small software modifications.

Aggressive Energy Reduction for Video Inference with Software-only Strategies

Given the saturation of single-threaded performance improvements in General-Purpose Processors (GPPs), novel architectural techniques are required to meet emerging demands. In this paper, we propose a generic acceleration framework for approximate algorithms that replaces computation with table look-up accesses in dedicated memories. At compile time, annotated application kernels are automatically profiled using sample inputs, and the most representative input-output mappings of each kernel are selected by using K-Means Clustering and saved in the program binary. At runtime, these mappings are loaded into dedicated look-up tables, and kernel execution is replaced by hardware execution of the Nearest-Centroid Classifier, which selects from memory the best matching output to the region. We show a comparison with a similar framework based on neural acceleration and that, under similar levels of quality, the proposed approach achieves on average three times better performance and energy with significant area savings, thus opening new opportunities for performance harvesting in approximate accelerators.

Efficient Local Memory Support for Approximate Computing

Embedded video-processing applications are everywhere, and need to be low-energy in order to extend battery life. Convolutional Neural Networks (CNNs), frequently used for this task, fail to explore the intrinsic redundancy present in videos: similarity between sequential frames means that analyzing all frames can be avoided. On top of that, while several hardware solutions for low-energy execution have been proposed, they require extra or dedicated hardware, which makes them non attractive for low cost applications. In this work we propose a technique that uses frame similarity to identify and process only areas that have a significant difference when comparing two subsequent frames. Our technique reduces energy consumption by discarding unneeded operations, and can also be used in low-cost hardware readily available for IoT applications. We obtain up to 12-80x speedup of CNN execution with software-only modifications that require no network retraining while impacting little on accuracy.

Larissa Rozales Goncalves

Papers

Employing classification-based algorithms for general-purpose approximate computing

Aggressive Energy Reduction for Video Inference with Software-only Strategies

Efficient Local Memory Support for Approximate Computing

Using Frame Similarity for Low Energy Software-Only IoT Video Recognition