Open AccessJournal Article
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Colin Raffel,Noam Shazeer,Adam Roberts,Katherine Lee,Sharan Narang,Michael Matena,Yanqi Zhou,Wei Li,Peter J. Liu +8 more
TLDR
This article introduced a unified framework that converts all text-based language problems into a text-to-text format and compared pre-training objectives, architectures, unlabeled data sets, transfer approaches, and other factors on dozens of language understanding tasks.Abstract:
Transfer learning, where a model is first pre-trained on a data-rich task before being fine-tuned on a downstream task, has emerged as a powerful technique in natural language processing (NLP). The effectiveness of transfer learning has given rise to a diversity of approaches, methodology, and practice. In this paper, we explore the landscape of transfer learning techniques for NLP by introducing a unified framework that converts all text-based language problems into a text-to-text format. Our systematic study compares pre-training objectives, architectures, unlabeled data sets, transfer approaches, and other factors on dozens of language understanding tasks. By combining the insights from our exploration with scale and our new ``Colossal Clean Crawled Corpus'', we achieve state-of-the-art results on many benchmarks covering summarization, question answering, text classification, and more. To facilitate future work on transfer learning for NLP, we release our data set, pre-trained models, and code.read more
Citations
More filters
Posted Content
NeurIPS 2020 NLC2CMD Competition: Translating Natural Language to Bash Commands
Mayank Agarwal,Tathagata Chakraborti,Quchen Fu,David Gros,Xi Victoria Lin,Jaron Maene,Kartik Talamadupula,Zhongwei Teng,Jules White +8 more
TL;DR: The NLC2CMD 2019 NLP challenge as mentioned in this paper aimed to bring the power of natural language processing to the command line and participants were tasked with building models that can transform descriptions of command line tasks in English to their Bash syntax.
Proceedings Article
Gated Transformer for Robust De-noised Sequence-to-Sequence Modelling.
TL;DR: This paper proposed a modified Transformer-based encoder-decoder architecture that uses a gating mechanism to detect types of corrections required and accordingly corrects texts, which shows that the gated architecture with pre-trained language models perform significantly better that the non-gated counterparts and other state-of-the-art error correction models in correcting spelling and grammatical errors.
Proceedings Article
Exploring Metaphoric Paraphrase Generation
TL;DR: The authors compare naive, "free" generation models with those that exploit forms of control over the generation process, adding additional information based on conceptual metaphor theory, showing that free models tend to generate more fluent paraphrases, while controlled models are better at generating novel metaphors.
Journal ArticleDOI
NAREOR: The Narrative Reordering Problem
TL;DR: This paper proposed and investigated the task of Narrative Reordering (NAREOR) which involves rewriting a given story in a different narrative order while preserving its plot, and presented a dataset, NAREORC, with human rewritings of stories within ROCStories in non-linear orders.
Posted Content
RiddleSense: Reasoning about Riddle Questions Featuring Linguistic Creativity and Commonsense Knowledge
TL;DR: RiddleSense as mentioned in this paper is a dataset for answering riddle-style commonsense questions with a large dataset of 5.7k examples, with a focus on natural language understanding.