Open AccessJournal Article
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Colin Raffel,Noam Shazeer,Adam Roberts,Katherine Lee,Sharan Narang,Michael Matena,Yanqi Zhou,Wei Li,Peter J. Liu +8 more
Reads0
Chats0
TLDR
This article introduced a unified framework that converts all text-based language problems into a text-to-text format and compared pre-training objectives, architectures, unlabeled data sets, transfer approaches, and other factors on dozens of language understanding tasks.Abstract:
Transfer learning, where a model is first pre-trained on a data-rich task before being fine-tuned on a downstream task, has emerged as a powerful technique in natural language processing (NLP). The effectiveness of transfer learning has given rise to a diversity of approaches, methodology, and practice. In this paper, we explore the landscape of transfer learning techniques for NLP by introducing a unified framework that converts all text-based language problems into a text-to-text format. Our systematic study compares pre-training objectives, architectures, unlabeled data sets, transfer approaches, and other factors on dozens of language understanding tasks. By combining the insights from our exploration with scale and our new ``Colossal Clean Crawled Corpus'', we achieve state-of-the-art results on many benchmarks covering summarization, question answering, text classification, and more. To facilitate future work on transfer learning for NLP, we release our data set, pre-trained models, and code.read more
Citations
More filters
Posted Content
ChemBERTa: Large-Scale Self-Supervised Pretraining for Molecular Property Prediction
TL;DR: This work makes one of the first attempts to systematically evaluate transformers on molecular property prediction tasks via the ChemBERTa model, and suggests that transformers offer a promising avenue of future work for molecular representation learning and property prediction.
Proceedings ArticleDOI
Syntactic Data Augmentation Increases Robustness to Inference Heuristics
TL;DR: The best-performing augmentation method, subject/object inversion, improved BERT’s accuracy on controlled examples that diagnose sensitivity to word order from 0.28 to 0.73, suggesting that augmentation causes BERT to recruit abstract syntactic representations.
Proceedings ArticleDOI
MinTL: Minimalist Transfer Learning for Task-Oriented Dialogue Systems
TL;DR: This paper introduces Levenshtein belief spans (Lev), that allows efficient dialogue state tracking with a minimal generation length, and greatly improves the inference efficiency of MinTL-based systems.
Proceedings ArticleDOI
ZeRO: Memory optimizations Toward Training Trillion Parameter Models
TL;DR: The Zero Redundancy Optimizer (ZeRO) as mentioned in this paper eliminates memory redundancies in data and model-parallel training while retaining low communication volume and high computational granularity, allowing to scale the model size proportional to the number of devices.
Proceedings ArticleDOI
Less is More: CLIPBERT for Video-and-Language Learning via Sparse Sampling
TL;DR: ClipBERT as mentioned in this paper employs sparse sampling, where only a single or a few sparsely sampled short clips from a video are used at each training step to enable affordable end-to-end learning for video and language tasks.