Rainbow Memory: Continual Learning with a Memory of Diverse Samples
Jihwan Bang,Heesu Kim,Youngjoon Yoo,Jung-Woo Ha,Jonghyun Choi +4 more
- pp 8218-8227
Reads0
Chats0
TLDR
Rainbow Memory as mentioned in this paper proposes a memory management strategy based on per-sample classification uncertainty and data augmentation, which significantly improves the accuracy in blurry continual learning setups, outperforming state of the arts by large margins despite its simplicity.Abstract:
Continual learning is a realistic learning scenario for AI models. Prevalent scenario of continual learning, however, assumes disjoint sets of classes as tasks and is less realistic rather artificial. Instead, we focus on ‘blurry’ task boundary; where tasks shares classes and is more realistic and practical. To address such task, we argue the importance of diversity of samples in an episodic memory. To enhance the sample diversity in the memory, we propose a novel memory management strategy based on per-sample classification uncertainty and data augmentation, named Rainbow Memory (RM). With extensive empirical validations on MNIST, CIFAR10, CIFAR100, and ImageNet datasets, we show that the proposed method significantly improves the accuracy in blurry continual learning setups, outperforming state of the arts by large margins despite its simplicity. Code and data splits will be available in https://github.com/clovaai/rainbow-memory.read more
Citations
More filters
Proceedings ArticleDOI
Constrained Few-shot Class-incremental Learning
Michael Hersche,Geethan Karunaratne,Giovanni Cherubini,Luca Benini,Abu Sebastian,Abbas Rahimi +5 more
TL;DR: C-FSCIL provides three update modes that offer a trade-off between accuracy and compute-memory cost of learning novel classes, and exploits hyperdimensional embedding that allows to continually express many more classes than the fixed dimensions in the vector space, with minimal interference.
Proceedings ArticleDOI
Learn from Others and Be Yourself in Heterogeneous Federated Learning
Wenke Huang,Mang Ye,Bo Du +2 more
TL;DR: This work proposes FCCL (Federated CrossCorrelation and Continual Learning), which leverages unlabeled public data for communication and construct cross-correlation matrix to learn a generalizable representation under domain shift for heterogeneity problem and catastrophic forgetting.
Proceedings ArticleDOI
Information-theoretic Online Memory Selection for Continual Learning
TL;DR: This work proposes the surprise and the learnability criteria to pick informative points and to avoid outliers, and presents a Bayesian model to compute the criteria efficiently by exploiting rank-one matrix structures.
Journal ArticleDOI
Knowledge extraction and retention based continual learning by using convolutional autoencoder-based learning classifier system
TL;DR: In this paper , a deep convolutional autoencoder is presented to extract features from images and a learning classifier system with an effective knowledge encoding scheme is proposed for mapping real-world images to code fragment-based knowledge representation.
Proceedings ArticleDOI
Continual Learning with Lifelong Vision Transformer
TL;DR: An inter-task attention mechanism is presented in LVT, which implicitly absorbs the previous tasks' information and slows down the drift of important attention between previous tasks and the current task to achieve a better stability-plasticity trade-off for continual learning.
References
More filters
Proceedings ArticleDOI
Deep Residual Learning for Image Recognition
TL;DR: In this article, the authors proposed a residual learning framework to ease the training of networks that are substantially deeper than those used previously, which won the 1st place on the ILSVRC 2015 classification task.
Proceedings Article
Dropout as a Bayesian approximation: representing model uncertainty in deep learning
Yarin Gal,Zoubin Ghahramani +1 more
TL;DR: A new theoretical framework is developed casting dropout training in deep neural networks (NNs) as approximate Bayesian inference in deep Gaussian processes, which mitigates the problem of representing uncertainty in deep learning without sacrificing either computational complexity or test accuracy.
Proceedings ArticleDOI
CutMix: Regularization Strategy to Train Strong Classifiers With Localizable Features
TL;DR: CutMix as discussed by the authors augments the training data by cutting and pasting patches among training images, where the ground truth labels are also mixed proportionally to the area of the patches.
Journal ArticleDOI
Overcoming catastrophic forgetting in neural networks
James Kirkpatrick,Razvan Pascanu,Neil C. Rabinowitz,Joel Veness,Guillaume Desjardins,Andrei Rusu,Kieran Milan,John Quan,Tiago Ramalho,Agnieszka Grabska-Barwinska,Demis Hassabis,Claudia Clopath,Dharshan Kumaran,Raia Hadsell +13 more
TL;DR: In this paper, the authors show that it is possible to train networks that can maintain expertise on tasks that they have not experienced for a long time by selectively slowing down learning on the weights important for those tasks.
Posted Content
Improved Regularization of Convolutional Neural Networks with Cutout.
TL;DR: This paper shows that the simple regularization technique of randomly masking out square regions of input during training, which is called cutout, can be used to improve the robustness and overall performance of convolutional neural networks.