Open AccessJournal Article
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Colin Raffel,Noam Shazeer,Adam Roberts,Katherine Lee,Sharan Narang,Michael Matena,Yanqi Zhou,Wei Li,Peter J. Liu +8 more
Reads0
Chats0
TLDR
This article introduced a unified framework that converts all text-based language problems into a text-to-text format and compared pre-training objectives, architectures, unlabeled data sets, transfer approaches, and other factors on dozens of language understanding tasks.Abstract:
Transfer learning, where a model is first pre-trained on a data-rich task before being fine-tuned on a downstream task, has emerged as a powerful technique in natural language processing (NLP). The effectiveness of transfer learning has given rise to a diversity of approaches, methodology, and practice. In this paper, we explore the landscape of transfer learning techniques for NLP by introducing a unified framework that converts all text-based language problems into a text-to-text format. Our systematic study compares pre-training objectives, architectures, unlabeled data sets, transfer approaches, and other factors on dozens of language understanding tasks. By combining the insights from our exploration with scale and our new ``Colossal Clean Crawled Corpus'', we achieve state-of-the-art results on many benchmarks covering summarization, question answering, text classification, and more. To facilitate future work on transfer learning for NLP, we release our data set, pre-trained models, and code.read more
Citations
More filters
Posted Content
Predicting Document Coverage for Relation Extraction.
TL;DR: In this article, the authors presented a dataset of 31,366 diverse documents for 520 entities and analyzed the correlation of document coverage with features like length, entity mention frequency, Alexa rank, language complexity and information retrieval scores.
Proceedings Article
Iconary: A Pictionary-Based Game for Testing Multimodal Communication with Drawings and Text
Christopher Clark,Jordi Salvador,Dustin Schwenk,Derrick Bonafilia,Mark Yatskar,Eric Kolve,Alvaro Herrasti,Jonghyun Choi,Sachin Mehta,Sam Skjonsberg,Carissa Schoenick,Aaron Sarnat,Hannaneh Hajishirzi,Aniruddha Kembhavi,Oren Etzioni,Ali Farhadi +15 more
TL;DR: In this article, a Guesser tries to identify a phrase that a Drawer is drawing by composing icons, and the Drawer iteratively revises the drawing to help the Guesser in response.
Proceedings Article
Abstract, Rationale, Stance: A Joint Model for Scientific Claim Verification
TL;DR: Zhang et al. as discussed by the authors proposed an approach, named ARSJoint, that jointly learns the modules for the three tasks with a machine reading comprehension framework by including claim information.
Journal ArticleDOI
Predict-Then-Decide: A Predictive Approach for Wait or Answer Task in Dialogue Systems
Zehao Lin,Shaobo Cui,Guodun Li,Xiaoming Kang,Feng Ji,Feng-Lin Li,Zhongzhou Zhao,Haiqing Chen,Yin Zhang +8 more
TL;DR: In this article, the authors propose a predictive approach named Predict-then-Decide (PTD) to solve the Wait-Or-Answer problem in a dialogue system, where the decision of decision model is made with the assistance of two ancillary prediction models: a user prediction and an agent prediction.
Posted Content
multiPRover: Generating Multiple Proofs for Improved Interpretability in Rule Reasoning
TL;DR: In this paper, the authors propose two variants of a proof set generation model, multi-label-multi-PRover and iterative-multiPRover, for reasoning over natural language rule bases.