Open AccessJournal Article
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Colin Raffel,Noam Shazeer,Adam Roberts,Katherine Lee,Sharan Narang,Michael Matena,Yanqi Zhou,Wei Li,Peter J. Liu +8 more
Reads0
Chats0
TLDR
This article introduced a unified framework that converts all text-based language problems into a text-to-text format and compared pre-training objectives, architectures, unlabeled data sets, transfer approaches, and other factors on dozens of language understanding tasks.Abstract:
Transfer learning, where a model is first pre-trained on a data-rich task before being fine-tuned on a downstream task, has emerged as a powerful technique in natural language processing (NLP). The effectiveness of transfer learning has given rise to a diversity of approaches, methodology, and practice. In this paper, we explore the landscape of transfer learning techniques for NLP by introducing a unified framework that converts all text-based language problems into a text-to-text format. Our systematic study compares pre-training objectives, architectures, unlabeled data sets, transfer approaches, and other factors on dozens of language understanding tasks. By combining the insights from our exploration with scale and our new ``Colossal Clean Crawled Corpus'', we achieve state-of-the-art results on many benchmarks covering summarization, question answering, text classification, and more. To facilitate future work on transfer learning for NLP, we release our data set, pre-trained models, and code.read more
Citations
More filters
Posted Content
CycleGT: Unsupervised Graph-to-Text and Text-to-Graph Generation via Cycle Training.
TL;DR: CycleGT as discussed by the authors is an unsupervised training method that can bootstrap from fully non-parallel graph and text data, and iteratively back translate between the two forms.
Proceedings ArticleDOI
RefSum: Refactoring Neural Summarization
Yixin Liu,Zi-Yi Dou,Pengfei Liu +2 more
TL;DR: This work highlights several limitations of previous methods and presents a new framework Refactor that provides a unified view of text summarization and summaries combination and can be directly used by other researchers as an off-the-shelf tool to achieve further performance improvements.
Posted Content
Learning Contextual Representations for Semantic Parsing with Generation-Augmented Pre-Training
Peng Shi,Patrick Ng,Zhiguo Wang,Henghui Zhu,Alexander Hanbo Li,Jun Wang,Cicero Nogueira dos Santos,Bing Xiang +7 more
TL;DR: Generative-Augmented Pre-training (GAP) as mentioned in this paper learns representations of natural language utterances and table schemas by leveraging generation models to generate pre-train data.
Posted Content
A Survey on Self-supervised Pre-training for Sequential Transfer Learning in Neural Networks.
TL;DR: An overview of the taxonomy for self-supervised learning and transfer learning is provided, and some prominent methods for designing pre-training tasks across different domains are highlighted.
Proceedings ArticleDOI
Generating Fact Checking Briefs
Angela Fan,Aleksandra Piktus,Fabio Petroni,Guillaume Wenzek,Marzieh Saeidi,Andreas Vlachos,Antoine Bordes,Sebastian Riedel +7 more
TL;DR: The authors investigate how to increase the accuracy and efficiency of fact checking by providing information about the claim before performing the check, in the form of natural language briefs, and develop QABriefer, a model that generates a set of questions conditioned on the claim, searches the web for evidence, and generates answers.