On Connections Between Regularizations for Improving DNN Robustness
TLDR
In this paper, the authors analyze the regularization terms proposed recently for improving the adversarial robustness of deep neural networks from a theoretical point of view and study possible connections between several effective methods, including input-gradient regularization, Jacobian regularisation, curvature regularization and a cross-Lipschitz functional.Abstract:
This paper analyzes regularization terms proposed recently for improving the adversarial robustness of deep neural networks (DNNs), from a theoretical point of view. Specifically, we study possible connections between several effective methods, including input-gradient regularization, Jacobian regularization, curvature regularization, and a cross-Lipschitz functional. We investigate them on DNNs with general rectified linear activations, which constitute one of the most prevalent families of models for image classification and a host of other machine learning applications. We shed light on essential ingredients of these regularizations and re-interpret their functionality. Through the lens of our study, more principled and efficient regularizations can possibly be invented in the near future.read more
Citations
More filters
Posted Content
On the Generalization Properties of Adversarial Training
Yue Xing,Qifan Song,Guang Cheng +2 more
TL;DR: Inspired by successes of the least absolute shrinkage and selection operator (LASSO), the L1 penalty is incorporated in the high dimensional adversarial learning, and it is shown that it leads to consistent adversarial robust estimation in both theory and numerical trials.
Posted Content
Yet Another Intermediate-Level Attack
Qizhang Li,Yiwen Guo,Hao Chen +2 more
TL;DR: In this article, a linear mapping of the intermediate-level discrepancies between adversarial inputs and their benign counterparts is proposed to predict the evoked adversarial loss, taking full advantage of the optimization procedure of multi-step baseline attacks.
Book ChapterDOI
Yet Another Intermediate-Level Attack
Qizhang Li,Yiwen Guo,Hao Chen +2 more
TL;DR: A novel method to enhance the black-box transferability of baseline adversarial examples by establishing a linear mapping of the intermediate-level discrepancies for predicting the evoked adversarial loss by taking full advantage of the optimization procedure of multi-step baseline attacks.
Journal ArticleDOI
Manifold adversarial training for supervised and semi-supervised learning.
TL;DR: The manifold adversarial training (MAT) as discussed by the authors is a regularization method for deep learning based on the manifold of latent representations, which is a superset of the discriminative feature learning approach called center loss.
Journal ArticleDOI
Adversarial symmetric GANs: Bridging adversarial samples and adversarial networks.
TL;DR: Adversarial symmetric GANs (AS-GANs) are developed that incorporate adversarial training of the discriminator on real samples into vanilla GAns, making adversarialTraining symmetrical, thereby stabilizing training and accelerating convergence.
References
More filters
Proceedings ArticleDOI
Deep Residual Learning for Image Recognition
TL;DR: In this article, the authors proposed a residual learning framework to ease the training of networks that are substantially deeper than those used previously, which won the 1st place on the ILSVRC 2015 classification task.
Journal Article
Dropout: a simple way to prevent neural networks from overfitting
TL;DR: It is shown that dropout improves the performance of neural networks on supervised learning tasks in vision, speech recognition, document classification and computational biology, obtaining state-of-the-art results on many benchmark data sets.
Proceedings Article
Neural Machine Translation by Jointly Learning to Align and Translate
TL;DR: It is conjecture that the use of a fixed-length vector is a bottleneck in improving the performance of this basic encoder-decoder architecture, and it is proposed to extend this by allowing a model to automatically (soft-)search for parts of a source sentence that are relevant to predicting a target word, without having to form these parts as a hard segment explicitly.
Proceedings Article
Rectified Linear Units Improve Restricted Boltzmann Machines
Vinod Nair,Geoffrey E. Hinton +1 more
TL;DR: Restricted Boltzmann machines were developed using binary stochastic hidden units that learn features that are better for object recognition on the NORB dataset and face verification on the Labeled Faces in the Wild dataset.
Proceedings ArticleDOI
Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification
TL;DR: In this paper, a Parametric Rectified Linear Unit (PReLU) was proposed to improve model fitting with nearly zero extra computational cost and little overfitting risk, which achieved a 4.94% top-5 test error on ImageNet 2012 classification dataset.