Virtual Adversarial Training: A Regularization Method for Supervised and Semi-Supervised Learning
Reads0
Chats0
TLDR
Virtual adversarial training (VAT) as discussed by the authors is a regularization method based on virtual adversarial loss, which is a measure of local smoothness of the conditional label distribution given input.Abstract:
We propose a new regularization method based on virtual adversarial loss: a new measure of local smoothness of the conditional label distribution given input. Virtual adversarial loss is defined as the robustness of the conditional label distribution around each input data point against local perturbation. Unlike adversarial training, our method defines the adversarial direction without label information and is hence applicable to semi-supervised learning. Because the directions in which we smooth the model are only “virtually” adversarial, we call our method virtual adversarial training (VAT). The computational cost of VAT is relatively low. For neural networks, the approximated gradient of virtual adversarial loss can be computed with no more than two pairs of forward- and back-propagations. In our experiments, we applied VAT to supervised and semi-supervised learning tasks on multiple benchmark datasets. With a simple enhancement of the algorithm based on the entropy minimization principle, our VAT achieves state-of-the-art performance for semi-supervised learning tasks on SVHN and CIFAR-10.read more
Citations
More filters
Posted Content
Batch Virtual Adversarial Training for Graph Convolutional Networks.
Zhijie Deng,Yinpeng Dong,Jun Zhu +2 more
TL;DR: Two algorithms are proposed, sample-based and optimization-based BVAT, which are suitable to promote the smoothness of the model for graph-structured data by either finding virtual adversarial perturbations for a subset of nodes far from each other or generating virtual adversaries for all nodes with an optimization process.
Posted Content
3D Semi-Supervised Learning with Uncertainty-Aware Multi-View Co-Training
Yingda Xia,Fengze Liu,Dong Yang,Jinzheng Cai,Lequan Yu,Zhuotun Zhu,Daguang Xu,Alan L. Yuille,Holger R. Roth +8 more
TL;DR: A novel framework, uncertainty-aware multi-view co-training (UMCT), to address semi-supervised learning on 3D data, such as volumetric data from medical imaging, and proposes an uncertainty-weighted label fusion mechanism to estimate the reliability of each view’s prediction with Bayesian deep learning.
Journal ArticleDOI
Semi Supervised Learning with Deep Embedded Clustering for Image Classification and Segmentation.
TL;DR: A flexible framework for semi-supervised learning that combines the power of supervised methods that learn feature representations using state-of-the-art deep convolutional neural networks with the deeply embedded clustering algorithm that assigns data points to clusters based on their probability distributions and feature representations learned by the networks.
Proceedings Article
Network Randomization: A Simple Technique for Generalization in Deep Reinforcement Learning
TL;DR: In this article, a randomized convolutional neural network that randomly perturbs input observations is proposed to improve the generalization ability of deep RL agents by introducing a randomized (convolutional) neural network, which enables trained agents to adapt to new domains by learning robust features invariant across varied and randomized environments.
Proceedings Article
Intriguing Properties of Adversarial Examples
TL;DR: In this paper, the authors argue that the origin of adversarial examples is primarily due to an inherent uncertainty that neural networks have about their predictions, and they show that the functional form of this uncertainty is independent of architecture, dataset, and training protocol.
References
More filters
Proceedings Article
Adam: A Method for Stochastic Optimization
Diederik P. Kingma,Jimmy Ba +1 more
TL;DR: This work introduces Adam, an algorithm for first-order gradient-based optimization of stochastic objective functions, based on adaptive estimates of lower-order moments, and provides a regret bound on the convergence rate that is comparable to the best known results under the online convex optimization framework.
Journal ArticleDOI
Generative Adversarial Nets
Ian Goodfellow,Jean Pouget-Abadie,Mehdi Mirza,Bing Xu,David Warde-Farley,Sherjil Ozair,Aaron Courville,Yoshua Bengio +7 more
TL;DR: A new framework for estimating generative models via an adversarial process, in which two models are simultaneously train: a generative model G that captures the data distribution and a discriminative model D that estimates the probability that a sample came from the training data rather than G.
Journal Article
Dropout: a simple way to prevent neural networks from overfitting
TL;DR: It is shown that dropout improves the performance of neural networks on supervised learning tasks in vision, speech recognition, document classification and computational biology, obtaining state-of-the-art results on many benchmark data sets.
Proceedings Article
Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
Sergey Ioffe,Christian Szegedy +1 more
TL;DR: Applied to a state-of-the-art image classification model, Batch Normalization achieves the same accuracy with 14 times fewer training steps, and beats the original model by a significant margin.
Proceedings ArticleDOI
Densely Connected Convolutional Networks
TL;DR: DenseNet as mentioned in this paper proposes to connect each layer to every other layer in a feed-forward fashion, which can alleviate the vanishing gradient problem, strengthen feature propagation, encourage feature reuse, and substantially reduce the number of parameters.