Channel pruning is one of the most important techniques for compressing neural networks with convolutional filters. However, in our study, we find strong similarities among some primary pruning criteria proposed in recent years. The sequence of filters'"importance" in a convolutional layer according to these criteria are almost the same, resulting in similar pruned structures. This finding can be explained by our assumption that the trained convolutional filters approximately follow a Gaussian-alike distribution, which is demonstrated through systematic and comprehensive statistical tests. Under this assumption, the similarity of these criteria is theoretically proved. Moreover, we also find that if the network has too much redundancy(exists a large number of filters in each convolutional layer), then these criteria can not distinguish the "importance" of the filters. This phenomenon is due to that the convolutional layer will form a special geometric structure when redundancy is large enough and our assumption holds: for every pair of filters in one layer, (1)Their $\ell_2$ norm are equivalent; (2)They are equidistant; (3)and they are orthogonal. The full appendix is released at this https URL.

Convolution-Weight-Distribution Assumption: Rethinking the Criteria of Channel Pruning.

Image restoration tasks have witnessed great performance improvement in recent years by developing large deep models. Despite the outstanding performance, the heavy computation demanded by the deep models has restricted the application of image restoration. To lift the restriction, it is required to reduce the size of the networks while maintaining accuracy. Recently, N:M structured pruning has appeared as one of the effective and practical pruning approaches for making the model efficient with the accuracy constraint. However, it fails to account for different computational complexities and performance requirements for different layers of an image restoration network. To further optimize the trade-off between the efficiency and the restoration accuracy, we propose a novel pruning method that determines the pruning ratio for N:M structured sparsity at each layer. Extensive experimental results on super-resolution and deblurring tasks demonstrate the efficacy of our method which outperforms previous pruning methods significantly. PyTorch implementation for the proposed methods will be publicly available at https://github.com/JungHunOh/SLS_CVPR2022

Attentive Fine-Grained Structured Sparsity for Image Restoration

The advancement of convolutional neural networks (CNNs) on various vision applications has attracted lots of attention. Yet the majority of CNNs are unable to satisfy the strict requirement for real-world deployment. To overcome this, the recent popular network pruning is an effective method to reduce the redundancy of the models. However, the ranking of filters according to their “importance” on different pruning criteria may be inconsistent. One filter could be important according to a certain criterion, while it is unnecessary according to another one, which indicates that each criterion is only a partial view of the comprehensive “importance”. From this motivation, we propose a novel framework to integrate the existing filter pruning criteria by exploring the criteria diversity. The proposed framework contains two stages: Criteria Clustering and Filters Importance Calibration. First, we condense the pruning criteria via layerwise clustering based on the rank of “importance” score. Second, within each cluster, we propose a calibration factor to adjust their significance for each selected blending candidates and search for the optimal blending criterion via Evolutionary Algorithm. Quantitative results on the CIFAR-100 and ImageNet benchmarks show that our framework outperforms the state-of-the-art baselines, regrading to the compact model performance after pruning.

Blending Pruning Criteria for Convolutional Neural Networks

—Recently many plug-and-play self-attention modules (SAMs) are proposed to enhance the model generalization by exploiting the internal information of deep convolutional neural networks (CNNs). In general, previous works ignore where to plug in the SAMs since they connect the SAMs individually with each block of the entire CNN backbone for granted, leading to incremental computational cost and the number of parameters with the growth of network depth. However, we empirically ﬁnd and verify some counterintuitive phenomena that: (a) Connecting the SAMs to all the blocks may not always bring the largest performance boost, and connecting to partial blocks would be even better; (b) Adding the SAMs to a CNN may not always bring a performance boost, and instead it may even harm the performance of the original CNN backbone. Therefore, we articulate and demonstrate the Lottery Ticket Hypothesis for Self-attention Networks: a full self-attention network contains a subnetwork with sparse self-attention con- nections that can (1) accelerate inference, (2) reduce extra parameter increment, and (3) maintain accuracy. In addition to the empirical evidence, this hypothesis is also supported by our theoretical evidence. Furthermore, we propose a simple yet effective reinforcement-learning-based method to search the ticket, i.e ., the connection scheme that satisﬁes the three above- mentioned conditions. Extensive experiments on widely-used benchmark datasets and popular self-attention networks show the effectiveness of our method. Besides, our experiments illustrate that our searched ticket has the capacity of transferring to some vision tasks, e.g. , crowd counting and segmentation.

The Lottery Ticket Hypothesis for Self-attention in Convolutional Neural Network

Genetic Programming (GP) is one of the evolutionary computation (EC) methods followed with great interest by many researchers. When GP first appeared, it has become a popular computational intelligence method because of its successful applications and its potentials to find effective solutions for difficult practical problems of many different disciplines. With the use of GP in a wide variety of areas, numerous variants of GP methods have emerged to provide more effective solutions for computation problems of diverse application fields. Therefore, GP has a very rich literature that is progressively growing. Many GP software tools developed along with process of GP algorithms. There is a need for an inclusive survey of GP literature from the beginning to today of GP in order to reveal the role of GP in the computational intelligence field. This survey study aims to provide an overview of the growing GP literature in a systematic way. The researchers, who need to implement GP methods, can gain insight of potentials in GP methods, their essential drawbacks and prevalent superiorities. Accordingly, taxonomy of GP methods is given by a systematic review of popular GP methods. In this manner, GP methods are analyzed according to two main categories, which consider the discrepancies in their program (chromosome) representation styles and their methodologies. Besides, GP applications in diverse problems are summarized. This literature survey is especially useful for new researchers to gain the required broad perspective before implementing a GP method in their problems.

/pdf/a-review-of-genetic-programming-popular-techniques-jsdn3ovzbc.pdf

A Review of Genetic Programming: Popular Techniques, Fundamental Aspects, Software Tools and Applications

Network pruning for deep convolutional neural networks (CNNs) has recently achieved notable research progress on image-level classification. However, most existing pruning methods are not catered to or evaluated on semantic segmentation networks. In this paper, we advocate the importance of contextual information during channel pruning for semantic segmentation networks by presenting a novel Context-aware Pruning framework. Concretely, we formulate the embedded contextual information by leveraging the layer-wise channels interdependency via the Context-aware Guiding Module (CAGM) and introduce the Context-aware Guided Sparsification (CAGS) to adaptively identify the informative channels on the cumbersome model by inducing channel-wise sparsity on the scaling factors in batch normalization (BN) layers. The resulting pruned models require significantly lesser operations for inference while maintaining comparable performance to (at times outperforming) the original models. We evaluated our framework on widely-used benchmarks and showed its effectiveness on both large and lightweight models. On Cityscapes dataset, our framework reduces the number of parameters by 32%, 47%, 54%, and 63%, on PSPNet101, PSPNet50, ICNet, and SegNet, respectively, while preserving the performance.

/pdf/cap-context-aware-pruning-for-semantic-segmentation-1didjhklrl.pdf

CAP: Context-Aware Pruning for Semantic Segmentation

Recently, many plug-and-play self-attention modules are proposed to enhance the model generalization by exploiting the internal information of deep convolutional neural networks (CNNs). Previous works lay an emphasis on the design of attention module for specific functionality, e.g., light-weighted or task-oriented attention. However, they ignore the importance of where to plug in the attention module since they connect the modules individually with each block of the entire CNN backbone for granted, leading to incremental computational cost and number of parameters with the growth of network depth. Thus, we propose a framework called Efficient Attention Network (EAN) to improve the efficiency for the existing attention modules. In EAN, we leverage the sharing mechanism (Huang et al. 2020) to share the attention module within the backbone and search where to connect the shared attention module via reinforcement learning. Finally, we obtain the attention network with sparse connections between the backbone and modules, while (1) maintaining accuracy (2) reducing extra parameter increment and (3) accelerating inference. Extensive experiments on widely-used benchmarks and popular attention networks show the effectiveness of EAN. Furthermore, we empirically illustrate that our EAN has the capacity of transferring to other tasks and capturing the informative features. The code is available at this https URL

Efficient Attention Network: Accelerate Attention by Searching Where to Plug.

ACSL: Adaptive correlation-driven sparsity learning for deep neural network compression.

The invention discloses a human body behavior recognition method based on self-feedback gene expression programming. According to the method, for a deep human body behavior image, three-dimensional time series data of a plurality of joints of the human body are extracted from the image as samples; modeling is carried out on the samples by using self-feedback gene expression programming constructed by adding a TIS insertion string operation into gene expression programming after intersection and variation, thereby obtaining a human body motion model, wherein the TIS insertion string operation indicates the operation of inserting a function string into a joint motion sequence head; and then gradient information is extracted as a model feature. The model characteristics of the human body motion model of the training sample are input to a neural network and training is carried out to obtain a neural network model as a human body behavior classifier; and model characteristic of a human body movement model corresponding to a tested sample are inputted into the obtained human body behavior classifier to obtain a human body behavior recognition result. The method has advantages of high human body behavior identification accuracy and fast identification speed.

Wei He

Papers

CAP: Context-Aware Pruning for Semantic Segmentation

Efficient Attention Network: Accelerate Attention by Searching Where to Plug.

Blending Pruning Criteria for Convolutional Neural Networks

ACSL: Adaptive correlation-driven sparsity learning for deep neural network compression.

Human body behavior recognition method based on self-feedback gene expression programming