Neural network pruning---the task of reducing the size of a network by removing parameters---has been the subject of a great deal of work in recent years. We provide a meta-analysis of the literature, including an overview of approaches to pruning and consistent findings in the literature. After aggregating results across 81 papers and pruning hundreds of models in controlled conditions, our clearest finding is that the community suffers from a lack of standardized benchmarks and metrics. This deficiency is substantial enough that it is hard to compare pruning techniques to one another or determine how much progress the field has made over the past three decades. To address this situation, we identify issues with current practices, suggest concrete remedies, and introduce ShrinkBench, an open-source framework to facilitate standardized evaluations of pruning methods. We use ShrinkBench to compare various pruning techniques and show that its comprehensive evaluation can prevent common pitfalls when comparing pruning methods.

What is the State of Neural Network Pruning

Advances in Neural Information Processing Systems 27

One of the biggest issues in deep learning theory is the generalization ability of networks with huge model size. The classical learning theory suggests that overparameterized models cause overfitting. However, practically used large deep models avoid overfitting, which is not well explained by the classical approaches. To resolve this issue, several attempts have been made. Among them, the compression based bound is one of the promising approaches. However, the compression based bound can be applied only to a compressed network, and it is not applicable to the non-compressed original network. In this paper, we give a unified frame-work that can convert compression based bounds to those for non-compressed original networks. The bound gives even better rate than the one for the compressed network by improving the bias term. By establishing the unified frame-work, we can obtain a data dependent generalization error bound which gives a tighter evaluation than the data independent ones.

/pdf/compression-based-bound-for-non-compressed-network-unified-jewh1rgejv.pdf

Compression based bound for non-compressed network: unified generalization error analysis of large compressible deep neural network

Non-intrusive load monitoring (NILM) or energy disaggregation refers to the task of estimating the appliance power consumption given the aggregate power consumption readings. Recent state-of-the-art neural networks based methods are computation and memory intensive, and thus not suitable to run on "edge devices". Recent research has proposed various methods to compress neural networks without significantly impacting accuracy. In this work, we study different neural network compression schemes and their efficacy on the state-of-the-art neural network NILM method. We additionally propose a multi-task learning-based architecture to compress models further. We perform an extensive evaluation of these techniques on two publicly available datasets and find that we can reduce the memory and compute footprint by a factor of up to 100 without significantly impacting predictive performance.

EdgeNILM: Towards NILM on Edge devices

Neural networks tend to achieve better accuracy with training if they are larger -- even if the resulting models are overparameterized. Nevertheless, carefully removing such excess parameters before, during, or after training may also produce models with similar or even improved accuracy. In many cases, that can be curiously achieved by heuristics as simple as removing a percentage of the weights with the smallest absolute value -- even though magnitude is not a perfect proxy for weight relevance. With the premise that obtaining significantly better performance from pruning depends on accounting for the combined effect of removing multiple weights, we revisit one of the classic approaches for impact-based pruning: the Optimal Brain Surgeon(OBS). We propose a tractable heuristic for solving the combinatorial extension of OBS, in which we select weights for simultaneous removal, as well as a systematic update of the remaining weights. Our selection method outperforms other methods under high sparsity, and the weight update is advantageous even when combined with the other methods.

The Combinatorial Brain Surgeon: Pruning Weights That Cancel One Another in Neural Networks

Compression techniques for deep neural network models are becoming very important for the efficient execution of high-performance deep learning systems on edge-computing devices. The concept of model compression is also important for analyzing the generalization error of deep learning, known as the compression-based error bound. However, there is still huge gap between a practically effective compression method and its rigorous background of statistical learning theory. To resolve this issue, we develop a new theoretical framework for model compression and propose a new pruning method called {\it spectral pruning} based on this framework. We define the ``degrees of freedom'' to quantify the intrinsic dimensionality of a model by using the eigenvalue distribution of the covariance matrix across the internal nodes and show that the compression ability is essentially controlled by this quantity. Moreover, we present a sharp generalization error bound of the compressed model and characterize the bias--variance tradeoff induced by the compression procedure. We apply our method to several datasets to justify our theoretical analyses and show the superiority of the the proposed method.

Spectral Pruning: Compressing Deep Neural Networks via Spectral Analysis and its Generalization Error

Spectral-Pruning: Compressing deep neural network via spectral analysis

/pdf/spectral-pruning-compressing-deep-neural-networks-via-1yj6v3j01p.pdf

Spectral Pruning: Compressing Deep Neural Networks via Spectral Analysis and its Generalization Error.

Automatic image description is one of the challenging tasks of image recognitions. However, there are image descriptions that contain some too specific phrases that cannot be judged only from appearance of images. In this paper, we propose a novel approach to collect general phrases for generating image descriptions. On the assumption that there are high frequency phrases related to an query image in the image descriptions of similar images, we select nouns and their attribute phrases from the image descriptions of similar images based on their frequency. In order to evaluate the relevance of our image description, we conduct comparative experiments with existing approaches. Our experimental results show that our image descriptions are short, concise and visually relevant to query images.

Automatic image description by using word-level features

Verification of deep learning (DL) systems in various situations is growing in importance as they are being used in wider areas than ever before. To prevent unintentional behavior of a DL system, it is important to feed an unfamiliar input to simulate situations in which a model in the DL system can behave abnormally. Such an input is different from outliers since we consider only statistical characteristics to detect outliers and the response of the model is not taken into account. This paper proposes a method of evaluating a complexity of an input without a label for a given trained model for analyzing a dataset to find inputs unfamiliar to the model and estimating the similarity between two datasets. Specifically, we use an output of neurons in the model as an embedded representation of data. We regard typical patterns in the representations of data in a training dataset as features the model obtained from the training dataset. We then calculate a complexity of an input as a distance between the representation of the input and combination of the features. The obtained complexity can be applied for multiple purposes such as test guiding by extracting inputs that are complex for the model from a dataset, analyzing a training dataset to find an inappropriate input, and measuring the similarity between two datasets using complexity distribution. Experimental evaluations showed the effectiveness of the complexity calculated using the proposed method in analyzing a dataset.

Shingo Horiuchi

Papers

Spectral Pruning: Compressing Deep Neural Networks via Spectral Analysis and its Generalization Error

Spectral-Pruning: Compressing deep neural network via spectral analysis

Spectral Pruning: Compressing Deep Neural Networks via Spectral Analysis and its Generalization Error.

Automatic image description by using word-level features

Model-based Data-Complexity Estimator for Deep Learning Systems