Graph-Based Similarity of Neural Network Representations

/pdf/adversarial-sample-detection-via-channel-pruning-1fravemxr9.pdf

Adversarial Sample Detection via Channel Pruning

Lightweight model design has become an important direction in the application of deep learning technology, pruning is an effective mean to achieve a large reduction in model parameters and FLOPs. The existing neural network pruning methods mostly start from the importance of parameters, and design parameter evaluation metrics to perform parameter pruning iteratively. These methods are not studied from the perspective of model topology, may be effective but not efficient, and requires completely different pruning for different datasets. In this paper, we study the graph structure of the neural network, and propose regular graph based pruning (RGP) to perform a one-shot neural network pruning. We generate a regular graph, set the node degree value of the graph to meet the pruning ratio, and reduce the average shortest path length of the graph by swapping the edges to obtain the optimal edge distribution. Finally, the obtained graph is mapped into a neural network structure to realize pruning. Experiments show that the average shortest path length of the graph is negatively correlated with the classification accuracy of the corresponding neural network, and the proposed RGP shows a strong precision retention capability with extremely high parameter reduction (more than 90%) and FLOPs reduction (more than 90%).

RGP: Neural Network Pruning through Its Regular Graph Structure

There are good arguments to support the claim that feature representations
eventually transition from general to specific in deep neural networks (DNNs),
but this transition remains relatively underexplored. In this work, we move a
tiny step towards understanding the transition of feature representations. We
first characterize this transition by analyzing the class separation in
intermediate layers, and next model the process of class separation as
community evolution in dynamic graphs. Then, we introduce modularity, a common
metric in graph theory, to quantify the evolution of communities. We find that
modularity tends to rise as the layer goes deeper, but descends or reaches a
plateau at particular layers. Through an asymptotic analysis, we show that
modularity can provide quantitative analysis of the transition of the feature
representations. With the insight on feature representations, we demonstrate
that modularity can also be used to identify and locate redundant layers in
DNNs, which provides theoretical guidance for layer pruning. Based on this
inspiring finding, we propose a layer-wise pruning method based on modularity.
Further experiments show that our method can prune redundant layers with
minimal impact on performance. The codes are available at
https://github.com/yaolu-zjut/Dynamic-Graphs-Construction.

Yao Lu

Papers

Graph-Based Similarity of Neural Network Representations

Adversarial Sample Detection via Channel Pruning

RGP: Neural Network Pruning through Its Regular Graph Structure

Graph Modularity: Towards Understanding the Cross-Layer Transition of Feature Representations in Deep Neural Networks.