Convolutional neural network compression technology plays an extremely important role in model transplantation and deployment, especially in mobile and embedded hardware platforms with small memory and low computing power, compression technology is even more critical. Convolutional neural network channel pruning technology has developed rapidly in recent years, and a number of excellent pruning algorithms have emerged. The channel pruning technology has gradually developed from the earliest static pruning to dynamic pruning, which adopts different pruning schemes for different inputs. However, the current dynamic pruning scheme needs to introduce multiple modules to predict the mask to prune the feature maps, and some schemes also introduce multiple hyperparameters in the loss function to balance the model accuracy and pruning rate, which leads to The model has difficulty converging during training. We propose a dynamic pruning method, each convolution structure configures a simple prediction module, and generating dynamic labels through the input's norm and similarity to guide the prediction module training, which will not bring new parameters to the loss function. We conducted related experiments on multiple models on the Cifar10 datasets. The experiments on ResNet56 show that our scheme is 1.3% higher than the most advanced scheme in terms of compression rate under the premise of the same accuracy. 

Research on Dynamic Labels in Network Pruning

Efficient Search of Comprehensively Robust Neural Architectures Via Multi-Fidelity Evaluation

Neural architecture search (NAS) has emerged as one successful technique to find robust deep neural network (DNN) architectures. However, most existing robustness evaluations in NAS only consider $l_{\infty}$ norm-based adversarial noises. In order to improve the robustness of DNN models against multiple types of noises, it is necessary to consider a comprehensive evaluation in NAS for robust architectures. But with the increasing number of types of robustness evaluations, it also becomes more time-consuming to find comprehensively robust architectures. To alleviate this problem, we propose a novel efficient search of comprehensively robust neural architectures via multi-fidelity evaluation (ES-CRNA-ME). Specifically, we first search for comprehensively robust architectures under multiple types of evaluations using the weight-sharing-based NAS method, including different $l_{p}$ norm attacks, semantic adversarial attacks, and composite adversarial attacks. In addition, we reduce the number of robustness evaluations by the correlation analysis, which can incorporate similar evaluations and decrease the evaluation cost. Finally, we propose a multi-fidelity online surrogate during optimization to further decrease the search cost. On the basis of the surrogate constructed by low-fidelity data, the online high-fidelity data is utilized to finetune the surrogate. Experiments on CIFAR10 and CIFAR100 datasets show the effectiveness of our proposed method.

Efficient Search of Comprehensively Robust Neural Architectures via Multi-fidelity Evaluation

FGPGAN: a finer-grained CNN pruning via generative adversarial network

Convolutional neural network compression technology plays an extremely important role in model transplantation and deployment, especially in mobile and embedded hardware platforms with small memory and low computing power, compression technology is even more critical. Convolutional neural network channel pruning technology has developed rapidly in recent years, and a number of excellent pruning algorithms have emerged. The channel pruning technology has gradually developed from the earliest static pruning to dynamic pruning, which adopts different pruning schemes for different inputs. However, the current dynamic pruning scheme needs to introduce multiple modules to predict the mask to prune the feature maps, and some schemes also introduce multiple hyperparameters in the loss function to balance the model accuracy and pruning rate, which leads to The model has difficulty converging during training. We propose a dynamic pruning method, each convolution structure configures a simple prediction module, and generating dynamic labels through the input's norm and similarity to guide the prediction module training, which will not bring new parameters to the loss function. We conducted related experiments on multiple models on the Cifar10 datasets. The experiments on ResNet56 show that our scheme is 1.3% higher than the most advanced scheme in terms of compression rate under the premise of the same accuracy.

Filter pruning is one of the most effective methods to compress deep convolutional networks (CNNs). In this paper, as a key component in filter pruning, We first propose a feature discrimination based filter importance criterion, namely Receptive Field Criterion (RFC). It turns the maximum activation responses that characterize the receptive field into probabilities, then measure the filter importance by the distribution of these probabilities from a new perspective of feature discrimination. However, directly applying RFC to global threshold pruning may lead to some problems, because global threshold pruning neglects the differences between different layers. Hence, we propose Distinguishing Layer Pruning based on RFC (DLRFC), i.e., discriminately prune the filters in different layers, which avoids measuring filters between different layers directly against filter criteria. Specifically, our method first selects relatively redundant layers by hard and soft changes of the network output, and then prunes only at these layers. The whole process dynamically adjusts redundant layers through iterations. Extensive experiments conducted on CIFAR-10/100 and ImageNet show that our method achieves state-of-the-art performance in several benchmarks. 

Filter Pruning via Feature Discrimination in Deep Neural Networks

AbstractThe vulnerability of Deep Neural Networks, i.e., susceptibility to adversarial attacks, severely limits the application of DNNs in security-sensitive domains. Most of existing methods improve model robustness from weight optimization, such as adversarial training. However, the architecture of DNNs is also a key factor to robustness, which is often neglected or underestimated. We propose Robust Network Architecture Search (RNAS) to obtain a robust network against adversarial attacks. We observe that an adversarial perturbation distorting the non-robust features in latent feature space can further aggravate misclassification. Based on this observation, we search the robust architecture through restricting feature distortion in the search process. Specifically, we define a network vulnerability metric based on feature distortion as a constraint in the search process. This process is modeled as a multi-objective bilevel optimization problem and a novel algorithm is proposed to solve this optimization. Extensive experiments conducted on CIFAR-10/100 and SVHN show that RNAS achieves the best robustness under various adversarial attacks compared with extensive baselines and SOTA methods.KeywordsAdversarial examplesNetwork architecture searchRoubst architecture 

Robust Network Architecture Search via Feature Distortion Restraining

Although Deep Neural Networks(DNNs) have achieved successful applications in many fields, they are vulnerable to adversarial examples.Adversarial training is one of the most effective methods to improve the robustness of DNNs, and it is generally considered as solving a saddle point problem that minimizes risk and maximizes perturbation.Therefore, powerful adversarial examples can effectively replicate the situation of perturbation maximization to solve the saddle point problem.The method proposed in this paper approximates the output of DNNs in the input neighborhood by using the Taylor expansion, and then optimizes it by using the Lagrange multiplier method to generate adversarial examples. If it is used for adversarial training, the DNNs can be effectively regularized and the defects of the model can be improved.

Wujie Zhou

Papers

Filter Pruning via Feature Discrimination in Deep Neural Networks

Robust Network Architecture Search via Feature Distortion Restraining

Towards Robust DNNs: An Taylor Expansion-Based Method for Generating Powerful Adversarial Examples.