Open AccessJournal Article
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Colin Raffel,Noam Shazeer,Adam Roberts,Katherine Lee,Sharan Narang,Michael Matena,Yanqi Zhou,Wei Li,Peter J. Liu +8 more
Reads0
Chats0
TLDR
This article introduced a unified framework that converts all text-based language problems into a text-to-text format and compared pre-training objectives, architectures, unlabeled data sets, transfer approaches, and other factors on dozens of language understanding tasks.Abstract:
Transfer learning, where a model is first pre-trained on a data-rich task before being fine-tuned on a downstream task, has emerged as a powerful technique in natural language processing (NLP). The effectiveness of transfer learning has given rise to a diversity of approaches, methodology, and practice. In this paper, we explore the landscape of transfer learning techniques for NLP by introducing a unified framework that converts all text-based language problems into a text-to-text format. Our systematic study compares pre-training objectives, architectures, unlabeled data sets, transfer approaches, and other factors on dozens of language understanding tasks. By combining the insights from our exploration with scale and our new ``Colossal Clean Crawled Corpus'', we achieve state-of-the-art results on many benchmarks covering summarization, question answering, text classification, and more. To facilitate future work on transfer learning for NLP, we release our data set, pre-trained models, and code.read more
Citations
More filters
Posted Content
B-PROP: Bootstrapped Pre-training with Representative Words Prediction for Ad-hoc Retrieval
TL;DR: Li et al. as mentioned in this paper proposed a bootstrapped pre-training method based on BERT for ad-hoc retrieval, which used the powerful contextual language model BERT to replace the classical unigram language model for the ROP task construction, and re-trained BERT itself towards the tailored objective for IR.
Proceedings Article
CDLM: Cross-Document Language Modeling
TL;DR: The authors introduced a new pretraining approach for multi-document language modeling, incorporating two key ideas into the masked language modeling self-supervised objective: first, instead of considering documents in isolation, they pretrain over sets of multiple related documents, encouraging the model to learn cross-document relationships.
Proceedings Article
Finetuning Pretrained Transformers into RNNs
Jungo Kasai,Hao Peng,Yizhe Zhang,Dani Yogatama,Gabriel Ilharco,Nikolaos Pappas,Yi Mao,Weizhu Chen,Noah A. Smith +8 more
TL;DR: The authors proposed a swap-then-finetune procedure to improve the performance of a pretrained transformer by replacing the softmax attention with a linear-complexity recurrent alternative and then finetuning.
Posted Content
A Simple and Effective Positional Encoding for Transformers
TL;DR: Decoupled Positional Attention for Transformers (DIET) as mentioned in this paper is a simple yet effective mechanism to encode position and segment information into the Transformer models and has faster training and inference time, while achieving competitive performance on GLUE, XTREME and WMT benchmarks.
Posted Content
Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation
TL;DR: This paper proposed Attention with Linear Biases (ALiBi), which biases the query-key attention scores with a term that is proportional to their distance, and showed that this method allows training a 1.3 billion parameter model on input sequences of length 1024 that extrapolates to input sequence of length 2048.