Open AccessJournal Article
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Colin Raffel,Noam Shazeer,Adam Roberts,Katherine Lee,Sharan Narang,Michael Matena,Yanqi Zhou,Wei Li,Peter J. Liu +8 more
Reads0
Chats0
TLDR
This article introduced a unified framework that converts all text-based language problems into a text-to-text format and compared pre-training objectives, architectures, unlabeled data sets, transfer approaches, and other factors on dozens of language understanding tasks.Abstract:
Transfer learning, where a model is first pre-trained on a data-rich task before being fine-tuned on a downstream task, has emerged as a powerful technique in natural language processing (NLP). The effectiveness of transfer learning has given rise to a diversity of approaches, methodology, and practice. In this paper, we explore the landscape of transfer learning techniques for NLP by introducing a unified framework that converts all text-based language problems into a text-to-text format. Our systematic study compares pre-training objectives, architectures, unlabeled data sets, transfer approaches, and other factors on dozens of language understanding tasks. By combining the insights from our exploration with scale and our new ``Colossal Clean Crawled Corpus'', we achieve state-of-the-art results on many benchmarks covering summarization, question answering, text classification, and more. To facilitate future work on transfer learning for NLP, we release our data set, pre-trained models, and code.read more
Citations
More filters
Proceedings ArticleDOI
Counterfactual Data Augmentation for Neural Machine Translation
TL;DR: This work creates augmented parallel translation corpora by generating (path-specific) counterfactual aligned phrases by interpreting language models and phrasal alignment causally by taking both context and alignment into account.
Proceedings ArticleDOI
Assessing Phrasal Representation and Composition in Transformers
Lang Yu,Allyson Ettinger +1 more
TL;DR: The authors found that phrase representation in pre-trained transformers relies heavily on word content, with little evidence of nuanced composition, and identified variations in phrase representation quality across models, layers, and representation types, and made corresponding recommendations for usage of representations from these models.
Posted Content
Query Reformulation using Query History for Passage Retrieval in Conversational Search
TL;DR: This work proposes two simple yet effective query reformulation approaches: historical query expansion (HQE) and neural transfer reformulation (NTR), and shows that fusing their output reduces the performance gap between the manually-rewritten and automatically-generated queries to 4 from 22 points when compared with the best CAsT submission.
Proceedings Article
On Learning Universal Representations Across Languages
TL;DR: This paper proposed Hierarchical Contrastive Learning (HiCTL) to learn universal representations for parallel sentences distributed in one or multiple languages and distinguish the semantically-related words from a shared cross-lingual vocabulary for each sentence.
Proceedings ArticleDOI
Discourse Probing of Pretrained Language Models
TL;DR: BART is found to be overall the best model at capturing discourse — but only in its encoder, with BERT performing surprisingly well as the baseline model.