Open AccessJournal Article
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Colin Raffel,Noam Shazeer,Adam Roberts,Katherine Lee,Sharan Narang,Michael Matena,Yanqi Zhou,Wei Li,Peter J. Liu +8 more
TLDR
This article introduced a unified framework that converts all text-based language problems into a text-to-text format and compared pre-training objectives, architectures, unlabeled data sets, transfer approaches, and other factors on dozens of language understanding tasks.Abstract:
Transfer learning, where a model is first pre-trained on a data-rich task before being fine-tuned on a downstream task, has emerged as a powerful technique in natural language processing (NLP). The effectiveness of transfer learning has given rise to a diversity of approaches, methodology, and practice. In this paper, we explore the landscape of transfer learning techniques for NLP by introducing a unified framework that converts all text-based language problems into a text-to-text format. Our systematic study compares pre-training objectives, architectures, unlabeled data sets, transfer approaches, and other factors on dozens of language understanding tasks. By combining the insights from our exploration with scale and our new ``Colossal Clean Crawled Corpus'', we achieve state-of-the-art results on many benchmarks covering summarization, question answering, text classification, and more. To facilitate future work on transfer learning for NLP, we release our data set, pre-trained models, and code.read more
Citations
More filters
Proceedings ArticleDOI
A combined rule-based and machine learning approach for automated GDPR compliance checking
Rajaa El Hamdani,Majd Mustapha,David Restrepo Amariles,Aurore Troussel,Sébastien Meeùs,Katsiaryna Krasnashchok +5 more
TL;DR: Li et al. as discussed by the authors proposed a document-centric approach to compliance checking in the data supply chain, where the first module relies on NLP to extract data practices from privacy policies, and the second module encodes GDPR rules to check the presence of mandatory information.
Journal ArticleDOI
Contextualized query expansion via unsupervised chunk selection for text retrieval
TL;DR: In this article, the authors proposed a query expansion model using unsupervised chunk selection, coined as BERT-QE, which leverages the strength of BERT model to select relevant text chunks from feedback documents in phase two and uses them for the final re-ranking in phase three.
Proceedings ArticleDOI
RiddleSense: Reasoning about Riddle Questions Featuring Linguistic Creativity and Commonsense Knowledge
TL;DR: RiddleSense as mentioned in this paper is a dataset for answering riddle-style commonsense questions with a large dataset of 5.7k examples, with a focus on natural language understanding.
Proceedings Article
JASS: Japanese-specific Sequence to Sequence Pre-training for Neural Machine Translation
TL;DR: JASS, Japanese-specific Sequence to Sequence, is proposed as a novel pre-training alternative to MASS for NMT involving Japanese as the source or target language and it is shown for the first time that joint MASS and JASS pre- training gives results that significantly surpass the individual methods indicating their complementary nature.
Posted Content
End-to-End Training of Neural Retrievers for Open-Domain Question Answering
Devendra Singh Sachan,Mostofa Patwary,Mohammad Shoeybi,Neel Kant,Wei Ping,William L. Hamilton,Bryan Catanzaro +6 more
TL;DR: This paper proposed an approach of unsupervised pre-training with the Inverse Cloze Task and masked salient spans, followed by supervised finetuning using question-context pairs, which leads to absolute gains of 2+ points over the previous best result in the top-20 retrieval accuracy on Natural Questions and TriviaQA datasets.