Open AccessJournal Article
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Colin Raffel,Noam Shazeer,Adam Roberts,Katherine Lee,Sharan Narang,Michael Matena,Yanqi Zhou,Wei Li,Peter J. Liu +8 more
TLDR
This article introduced a unified framework that converts all text-based language problems into a text-to-text format and compared pre-training objectives, architectures, unlabeled data sets, transfer approaches, and other factors on dozens of language understanding tasks.Abstract:
Transfer learning, where a model is first pre-trained on a data-rich task before being fine-tuned on a downstream task, has emerged as a powerful technique in natural language processing (NLP). The effectiveness of transfer learning has given rise to a diversity of approaches, methodology, and practice. In this paper, we explore the landscape of transfer learning techniques for NLP by introducing a unified framework that converts all text-based language problems into a text-to-text format. Our systematic study compares pre-training objectives, architectures, unlabeled data sets, transfer approaches, and other factors on dozens of language understanding tasks. By combining the insights from our exploration with scale and our new ``Colossal Clean Crawled Corpus'', we achieve state-of-the-art results on many benchmarks covering summarization, question answering, text classification, and more. To facilitate future work on transfer learning for NLP, we release our data set, pre-trained models, and code.read more
Citations
More filters
Posted Content
BERMo: What can BERT learn from ELMo?
Sangamesh Kodge,Kaushik Roy +1 more
TL;DR: This article proposed BERMo, an architectural modification to BERT, which makes predictions based on a hierarchy of surface, syntactic and semantic language features and uses linear combination scheme proposed in Embeddings from Language Models (ELMo) to combine the scaled internal representations from different network depths.
Posted Content
NxMTransformer: Semi-Structured Sparsification for Natural Language Understanding via ADMM.
TL;DR: NxMTransformer as discussed by the authors proposes to formulate the NxM sparsity as a constrained optimization problem and use Alternating Direction Method of Multipliers (ADMM) to optimize the downstream tasks while taking the underlying hardware constraints into consideration.
Posted Content
Text Detoxification using Large Pre-trained Neural Models.
David Dale,Anton Voronov,Daryna Dementieva,Varvara Logacheva,Olga Kozlova,Nikita Semenov,Alexander Panchenko +6 more
TL;DR: This paper used paraphrasing models to remove toxicity in text and used BERT to replace toxic words with their non-offensive synonyms, making the method more flexible by enabling BERT with a variable number of words.
Posted Content
On the Challenges of Evaluating Compositional Explanations in Multi-Hop Inference: Relevance, Completeness, and Expert Ratings
TL;DR: This paper constructed a large corpus of 126k domain-expert relevance ratings that augment a corpus of explanations to standardized science exam questions, discovering 80k additional relevant facts not rated as gold.
Posted Content
TextBox: A Unified, Modularized, and Extensible Framework for Text Generation
Junyi Li,Tianyi Tang,Gaole He,Jinhao Jiang,Xiaoxuan Hu,Puzhao Xie,Zhipeng Chen,Zhuohao Yu,Wayne Xin Zhao,Ji-Rong Wen +9 more
TL;DR: TextBox as discussed by the authors provides a unified, modularized, and extensible text generation framework for a broad set of text generation tasks and models, including VAE, GAN, and pre-trained language models.