Artificial intelligence (AI) has achieved remarkable breakthroughs in a wide range of fields, ranging from speech processing, image classification to drug discovery. This is driven by the explosive growth of data, advances in machine learning (especially deep learning), and the easy access to powerful computing resources. Particularly, the wide scale deployment of edge devices (e.g., IoT devices) generates an unprecedented scale of data, which provides the opportunity to derive accurate models and develop various intelligent applications at the network edge. However, such enormous data cannot all be sent to the cloud for processing, due to the varying channel quality, traffic congestion and/or privacy concerns, and the enormous energy consumption. By pushing inference and training processes of AI models to edge nodes, edge AI has emerged as a promising alternative. AI at the edge requires close cooperation among edge devices , such as smart phones and smart vehicles, and edge servers at the wireless access points and base stations, which however result in heavy communication overheads. In this paper, we present a comprehensive survey of the recent developments in various techniques for overcoming these communication challenges. Specifically, we first identify key communication challenges in edge AI systems. We then introduce communication-efficient techniques, from both algorithmic and system perspectives for training and inference tasks at the network edge. Potential future research directions are also highlighted.

Communication-Efficient Edge AI: Algorithms and Systems

Artificial intelligence (AI) has achieved remarkable breakthroughs in a wide range of fields, ranging from speech processing, image classification to drug discovery. This is driven by the explosive growth of data, advances in machine learning (especially deep learning), and easy access to vastly powerful computing resources. Particularly, the wide scale deployment of edge devices (e.g., IoT devices) generates an unprecedented scale of data, which provides the opportunity to derive accurate models and develop various intelligent applications at the network edge. However, such enormous data cannot all be sent from end devices to the cloud for processing, due to the varying channel quality, traffic congestion and/or privacy concerns. By pushing inference and training processes of AI models to edge nodes, edge AI has emerged as a promising alternative. AI at the edge requires close cooperation among edge devices, such as smart phones and smart vehicles, and edge servers at the wireless access points and base stations, which however result in heavy communication overheads. In this paper, we present a comprehensive survey of the recent developments in various techniques for overcoming these communication challenges. Specifically, we first identify key communication challenges in edge AI systems. We then introduce communication-efficient techniques, from both algorithmic and system perspectives for training and inference tasks at the network edge. Potential future research directions are also highlighted.

We consider relative error low rank approximation of tensors with respect to the Frobenius norm. Namely, given an order- q tensor A ∈ RΠqi=1ni, output a rank-k tensor B for which ||A − B||2F ≤ (1 + ϵ) OPT, where OPT = infrank-kA' ||A − A'||2F. Despite much success on obtaining relative error low rank approximations for matrices, no such results were known for tensors for arbitrary (1 + ϵ)-approximations. One structural issue is that there may be no rank-k tensor Ak achieving the above infinum. Another, computational issue, is that an efficient relative error low rank approximation algorithm for tensors would allow one to compute the rank of a tensor, which is NP-hard. We bypass these two issues via (1) bicriteria and (2) parameterized complexity solutions:1. We give an algorithm which outputs a rank k' = O((k/ϵ)q − 1) tensor B for which ||A − B||2F ≤ (1 + ϵ) OPT in nnz(A) + n · poly(k/ϵ) time in the real RAM model, whenever either Ak exists or OPT > 0. Here nnz(A) denotes the number of non-zero entries in A. If both Ak does not exist and OPT = 0, then B instead satisfies ||A − B||2F 0 which outputs a rank k tensor B for which ||A − B||2F ≤ (1 + ϵ) OPT and runs in (nnz(A)+ n poly(k/ϵ)+exp(k2/ϵ))·nδ time in the unit cost RAM model, whenever OPT > 2−O(nδ) and there is a rank-k tensor B = ∑ki = 1ui ⊗ ui ⊗ wi for which ||A − B||2F ≤ (1 + ϵ/2) OPT and ||ui ||2, ||ui ||2, ||wi||2 ≤ 2O(nδ). If OPT ≤ 2-Ω(nδ), then B instead satisfies ||A − B||2F ≤ 2-Ω(nδ).Our first result is polynomial time, and in fact input sparsity time, in n, k, and 1/ϵ, for any k ≥ 1 and any 0 1, we show a 2Ω(k1 − o(1)) time lower bound under the Exponential Time Hypothesis.Our results are based on an "iterative existential argument", and also give the first relative error low rank approximations for tensors for a large number of error measures for which nothing was known. In particular, we give the first relative error approximation algorithms on tensors for: column row and tube subset selection, entrywise ep-low rank approximation for 1 ≤ p

Relative Error Tensor Low Rank Approximation.

Graph Neural Networks (GNNs) are widely applied to graph learning problems 1 such as node classiﬁcation. When scaling up the underlying graphs of GNNs to a 2 larger size, we are forced to either train on the complete graph and keep the full 3 graph adjacency and node embeddings in memory (which is often infeasible) or 4 mini-batch sample the graph (which results in exponentially growing computational 5 complexities with respect to the number of GNN layers). Various sampling-based 6 and historical-embedding-based methods are proposed to avoid this exponential 7 growth of complexities. However, none of these solutions eliminates the linear 8 dependence on graph size. This paper proposes a sketch-based algorithm whose 9 training time and memory grow sublinearly with respect to graph size by training 10 GNNs atop a few compact sketches of graph adjacency and node embeddings. 11 Based on polynomial tensor-sketch (PTS) theory, our framework provides a novel 12 protocol for sketching non-linear activations and graph convolution matrices in 13 GNNs, as opposed to existing methods that sketch linear weights or gradients 14 in neural networks. In addition, we develop a locality sensitive hashing (LSH) 15 technique that can be trained to improve the quality of sketches. Experiments on 16 large-graph benchmarks demonstrate the scalability and competitive performance 17 of our Sketch-GNNs versus their full-size GNN counterparts. 18

Sketch-GNN: Scalable Graph Neural Networks with Sublinear Training Complexity

There has recently been an increasing desire to evaluate neural networks locally on computationally-limited devices in order to exploit their recent effectiveness for several applications; such effectiveness has nevertheless come together with a considerable increase in the size of modern neural networks, which constitute a major downside in several of the aforementioned computationally-limited settings. There has thus been a demand of compression techniques for neural networks. Several proposal in this direction have been made, which famously include hashing-based methods and pruning-based ones. However, the evaluation of the efficacy of these techniques has so far been heterogeneous, with no clear evidence in favor of any of them over the others. The goal of this work is to address this latter issue by providing a comparative study. While most previous studies test the capability of a technique in reducing the number of parameters of state-of-the-art networks , we follow [CWT + 15] in evaluating their performance on basic ar-chitectures on the MNIST dataset and variants of it, which allows for a clearer analysis of some aspects of their behavior. To the best of our knowledge, we are the first to directly compare famous approaches such as HashedNet, Optimal Brain Damage (OBD), and magnitude-based pruning with L1 and L2 regularization among them and against equivalent-size feed-forward neural networks with simple (fully-connected) and structural (convolutional) neural networks. Rather surprisingly, our experiments show that (iterative) pruning-based methods are substantially better than the HashedNet architecture, whose compression doesn't appear advantageous to a carefully chosen convolutional network. We also show that, when the compression level is high, the famous OBD pruning heuristics deteriorates to the point of being less efficient than simple magnitude-based techniques.

A Comparative Study of Neural Network Compression

Parameter reduction has been a popular topic in deep learning due to the ever-
increasing size of deep neural network models and the need to train and run deep neural nets on resource limited machines. Despite many efforts in this area, there were no rigorous theoretical guarantees on why existing neural net compression methods should work. In this paper, we provide provable guarantees on some hashing-based parameter reduction methods in neural nets. First, we introduce a neural net compression scheme based on random linear sketching (which is usually implemented efficiently via hashing), and show that the sketched (smaller) network is able to approximate the original network on all
input data coming from any smooth well-conditioned low-dimensional manifold. The sketched network can also be trained directly via back-propagation. Next, we study the previously proposed HashedNets architecture and show that the optimization landscape of one-hidden-layer HashedNets has a local strong convexity property similar to a normal fully connected neural network. Together
with the initialization algorithm developed in [51], this implies that the parameters in HashedNets can be provably recovered. We complement our theoretical results with some empirical verification.

http://proceedings.mlr.press/v89/lin19a/lin19a.pdf

Towards a Theoretical Understanding of Hashing-Based Neural Nets

Parameter reduction has been an important topic in deep learning due to the ever-increasing size of deep neural network models and the need to train and run them on resource limited machines. Despite many efforts in this area, there were no rigorous theoretical guarantees on why existing neural net compression methods should work. In this paper, we provide provable guarantees on some hashing-based parameter reduction methods in neural nets. First, we introduce a neural net compression scheme based on random linear sketching (which is usually implemented efficiently via hashing), and show that the sketched (smaller) network is able to approximate the original network on all input data coming from any smooth and well-conditioned low-dimensional manifold. The sketched network can also be trained directly via back-propagation. Next, we study the previously proposed HashedNets architecture and show that the optimization landscape of one-hidden-layer HashedNets has a local strong convexity property similar to a normal fully connected neural network. We complement our theoretical results with empirical verifications.

Zhao Song

Papers

Towards a Theoretical Understanding of Hashing-Based Neural Nets

Towards a Theoretical Understanding of Hashing-Based Neural Nets