Open AccessProceedings Article
Regularizing Deep Neural Networks by Noise: Its Interpretation and Optimization
Hyeonwoo Noh,Tackgeun You,Jonghwan Mun,Bohyung Han +3 more
- Vol. 30, pp 5109-5118
Reads0
Chats0
TLDR
This paper interprets that the conventional training methods with regularization by noise injection optimize the lower bound of the true objective and proposes a technique to achieve a tighter lower bound using multiple noise samples per training example in a stochastic gradient descent iteration.Abstract:
Overfitting is one of the most critical challenges in deep neural networks, and there are various types of regularization methods to improve generalization performance. Injecting noises to hidden units during training, e.g., dropout, is known as a successful regularizer, but it is still not clear enough why such training techniques work well in practice and how we can maximize their benefit in the presence of two conflicting objectives---optimizing to true data distribution and preventing overfitting by regularization. This paper addresses the above issues by 1) interpreting that the conventional training methods with regularization by noise injection optimize the lower bound of the true objective and 2) proposing a technique to achieve a tighter lower bound using multiple noise samples per training example in a stochastic gradient descent iteration. We demonstrate the effectiveness of our idea in several computer vision applications.read more
Citations
More filters
Posted Content
Benchmarking Inference Performance of Deep Learning Models on Analog Devices
Omobayode Fagbohungbe,Lijun Qian +1 more
TL;DR: In this study, systematic evaluation of the inference performance of trained popular deep learning models for image classification deployed on analog devices has been carried out, where additive white Gaussian noise has been added to the weights of the trained models during inference.
Posted Content
Panda: AdaPtive Noisy Data Augmentation for Regularization of Undirected Graphical Models.
TL;DR: An AdaPtive Noise Augmentation technique to regularize the estimation and construction of undirected graphical models and derive the asymptotic distributions for the regularized parameters through PANDA in generalized linear models, based on which, inferences for the parameters can be obtained simultaneously with variable selection.
Journal ArticleDOI
Implicit adversarial data augmentation and robustness with Noise-based Learning
Priyadarshini Panda,Kaushik Roy +1 more
TL;DR: In this paper, the authors introduce a Noise-based Learning (NoL) approach for training neural networks that are intrinsically robust to adversarial attacks, where the learning of random noise introduced with the input with the same loss function used during posterior maximization, improves a model's adversarial resistance.
Posted Content
Learning to Generate Noise for Multi-Attack Robustness
TL;DR: In this paper, the authors propose a meta-learning framework that explicitly learns to generate noise to improve the model's robustness against multiple types of adversarial attacks, such as multiple adversaries adopting different adversaries to deceive the system.
Proceedings ArticleDOI
Infinite Dropout for training Bayesian models from data streams
TL;DR: The ability to reduce overfitting and the ensemble property of Dropout, the framework obtains better generalization, thus effectively handles undesirable effects of noise and sparsity and significantly outperforms the state-of-the-art baselines.
References
More filters
Proceedings ArticleDOI
Deep Residual Learning for Image Recognition
TL;DR: In this article, the authors proposed a residual learning framework to ease the training of networks that are substantially deeper than those used previously, which won the 1st place on the ILSVRC 2015 classification task.
Proceedings Article
ImageNet Classification with Deep Convolutional Neural Networks
TL;DR: The state-of-the-art performance of CNNs was achieved by Deep Convolutional Neural Networks (DCNNs) as discussed by the authors, which consists of five convolutional layers, some of which are followed by max-pooling layers, and three fully-connected layers with a final 1000-way softmax.
Proceedings Article
Very Deep Convolutional Networks for Large-Scale Image Recognition
Karen Simonyan,Andrew Zisserman +1 more
TL;DR: In this paper, the authors investigated the effect of the convolutional network depth on its accuracy in the large-scale image recognition setting and showed that a significant improvement on the prior-art configurations can be achieved by pushing the depth to 16-19 layers.
Journal Article
Dropout: a simple way to prevent neural networks from overfitting
TL;DR: It is shown that dropout improves the performance of neural networks on supervised learning tasks in vision, speech recognition, document classification and computational biology, obtaining state-of-the-art results on many benchmark data sets.
Posted Content
Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
TL;DR: Faster R-CNN as discussed by the authors proposes a Region Proposal Network (RPN) to generate high-quality region proposals, which are used by Fast R-NN for detection.