Open AccessJournal Article
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Colin Raffel,Noam Shazeer,Adam Roberts,Katherine Lee,Sharan Narang,Michael Matena,Yanqi Zhou,Wei Li,Peter J. Liu +8 more
TLDR
This article introduced a unified framework that converts all text-based language problems into a text-to-text format and compared pre-training objectives, architectures, unlabeled data sets, transfer approaches, and other factors on dozens of language understanding tasks.Abstract:
Transfer learning, where a model is first pre-trained on a data-rich task before being fine-tuned on a downstream task, has emerged as a powerful technique in natural language processing (NLP). The effectiveness of transfer learning has given rise to a diversity of approaches, methodology, and practice. In this paper, we explore the landscape of transfer learning techniques for NLP by introducing a unified framework that converts all text-based language problems into a text-to-text format. Our systematic study compares pre-training objectives, architectures, unlabeled data sets, transfer approaches, and other factors on dozens of language understanding tasks. By combining the insights from our exploration with scale and our new ``Colossal Clean Crawled Corpus'', we achieve state-of-the-art results on many benchmarks covering summarization, question answering, text classification, and more. To facilitate future work on transfer learning for NLP, we release our data set, pre-trained models, and code.read more
Citations
More filters
Posted Content
Primer: Searching for Efficient Transformers for Language Modeling
TL;DR: In this paper, the authors identify an architecture, named Primer, that has a smaller training cost than the original Transformer and other variants for auto-regressive language modeling, which can be mostly attributed to two simple modifications: squaring ReLU activations and adding a depthwise convolution layer after each Q, K, and V projection in self-attention.
Posted Content
Syntax-BERT: Improving Pre-trained Transformers with Syntax Trees
TL;DR: This paper proposed a framework named Syntax-BERT to incorporate the syntax trees effectively and efficiently into pre-trained Transformers for natural language understanding tasks and achieved consistent improvement over BERT, RoBERTa, and T5.
Posted Content
Controllable Semantic Parsing via Retrieval Augmentation
TL;DR: ControllAble Semantic Parser via Exemplar Retrieval (CASPER) as mentioned in this paper retrieves related exemplars from a retrieval index, augments them to the query, and then applies a generative seq2seq model to produce an output parse.
Proceedings ArticleDOI
Dict-BERT: Enhancing Language Model Pre-training with Dictionary
TL;DR: The authors proposed a self-supervised pre-training task on word and sentence-level alignment between input text sequence and rare word definitions to enhance language modeling representation with dictionary, which can significantly improve the understanding of rare words and boost model performance on various NLP downstream tasks.
Posted Content
IndT5: A Text-to-Text Transformer for 10 Indigenous Languages.
TL;DR: IndT5 as mentioned in this paper is the first Transformer language model for low-resource and Indigenous languages and is trained on the IndCorpus dataset for ten Indigenous languages (Indigenous languages and Spanish) as part of the AmericasNLP 2021 Shared Task on Open Machine Translation.