Open AccessJournal Article
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Colin Raffel,Noam Shazeer,Adam Roberts,Katherine Lee,Sharan Narang,Michael Matena,Yanqi Zhou,Wei Li,Peter J. Liu +8 more
Reads0
Chats0
TLDR
This article introduced a unified framework that converts all text-based language problems into a text-to-text format and compared pre-training objectives, architectures, unlabeled data sets, transfer approaches, and other factors on dozens of language understanding tasks.Abstract:
Transfer learning, where a model is first pre-trained on a data-rich task before being fine-tuned on a downstream task, has emerged as a powerful technique in natural language processing (NLP). The effectiveness of transfer learning has given rise to a diversity of approaches, methodology, and practice. In this paper, we explore the landscape of transfer learning techniques for NLP by introducing a unified framework that converts all text-based language problems into a text-to-text format. Our systematic study compares pre-training objectives, architectures, unlabeled data sets, transfer approaches, and other factors on dozens of language understanding tasks. By combining the insights from our exploration with scale and our new ``Colossal Clean Crawled Corpus'', we achieve state-of-the-art results on many benchmarks covering summarization, question answering, text classification, and more. To facilitate future work on transfer learning for NLP, we release our data set, pre-trained models, and code.read more
Citations
More filters
Posted Content
Few-Shot Bot: Prompt-Based Learning for Dialogue Systems.
TL;DR: This paper proposed a prompt-based few-shot learning method to learn conversational skills using a few examples in the context as the only source of learning, which achieves competitive performance to fully trained state-of-the-art models.
Posted Content
Tackling Multi-Answer Open-Domain Questions via a Recall-then-Verify Framework.
Zhihong Shao,Minlie Huang +1 more
TL;DR: This paper propose a recall-then-verify framework, which separates the reasoning process of each answer so that they can make better use of retrieved evidence while also leveraging the power of large models under the same memory constraint.
Posted Content
ForecastQA: A Question Answering Challenge for Event Forecasting with Temporal Text Data
TL;DR: In this paper, a restricted-domain, multiple-choice, question-answering (QA) task is formulated for event forecasting on temporal news documents, and the problem is formulated as a restricted domain, multiplechoice, QA task.
Posted Content
Attention-guided Generative Models for Extractive Question Answering
TL;DR: This article proposed a cross-attention-based approach to obtain an extractive answer span from the generative model by leveraging the decoder's crossattention patterns, and applied joint training to further improve QA performance.
Posted Content
From Paraphrasing to Semantic Parsing: Unsupervised Semantic Parsing via Synchronous Semantic Decoding
Shan Wu,Bo Chen,Chunlei Xin,Xianpei Han,Le Sun,Weipeng Zhang,Jiansong Chen,Fan Yang,Xunliang Cai +8 more
TL;DR: In this paper, an unsupervised semantic parsing method called Synchronous Semantic Decoding (SSD) is proposed, which can simultaneously resolve the semantic gap and the structure gap by jointly leveraging paraphrasing and grammar constrained decoding.