Open AccessJournal Article
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Colin Raffel,Noam Shazeer,Adam Roberts,Katherine Lee,Sharan Narang,Michael Matena,Yanqi Zhou,Wei Li,Peter J. Liu +8 more
TLDR
This article introduced a unified framework that converts all text-based language problems into a text-to-text format and compared pre-training objectives, architectures, unlabeled data sets, transfer approaches, and other factors on dozens of language understanding tasks.Abstract:
Transfer learning, where a model is first pre-trained on a data-rich task before being fine-tuned on a downstream task, has emerged as a powerful technique in natural language processing (NLP). The effectiveness of transfer learning has given rise to a diversity of approaches, methodology, and practice. In this paper, we explore the landscape of transfer learning techniques for NLP by introducing a unified framework that converts all text-based language problems into a text-to-text format. Our systematic study compares pre-training objectives, architectures, unlabeled data sets, transfer approaches, and other factors on dozens of language understanding tasks. By combining the insights from our exploration with scale and our new ``Colossal Clean Crawled Corpus'', we achieve state-of-the-art results on many benchmarks covering summarization, question answering, text classification, and more. To facilitate future work on transfer learning for NLP, we release our data set, pre-trained models, and code.read more
Citations
More filters
Proceedings ArticleDOI
Optum at MEDIQA 2021: Abstractive Summarization of Radiology Reports using simple BART Finetuning
TL;DR: In this article, transfer learning from pre-trained transformer models was used for abstractive report summarization in the BioNLP MEDIQA 2019 Task 3: Radiology Report Summarization.
Proceedings Article
Explainable Natural Language to Bash Translation using Abstract Syntax Tree
TL;DR: In this article, a transformer-based approach was proposed to generate Bash commands from natural language invocations with explanations using Abstract Syntax Trees (ASTs) and manual pages, which performs better than fine-tuned T5 and Seq2Seq models.
Posted Content
GLUCOSE: GeneraLized and COntextualized Story Explanations
Nasrin Mostafazadeh,Aditya Kalyanpur,Lori Moon,David W. Buchanan,Lauren Berkowitz,Or Biran,Jennifer Chu-Carroll +6 more
TL;DR: GLUCOSE as discussed by the authors is a large-scale dataset of implicit commonsense causal knowledge encoded as causal mini-theories about the world, each grounded in a narrative context, focusing on events, states, motivations, and emotions.
Posted Content
On the ability of monolingual models to learn language-agnostic representations.
TL;DR: This paper showed that monolingual models pretrained and finetuned on different languages achieve competitive performance compared to the ones that use the same target language, regardless of the pretraining language.
Journal ArticleDOI
Multistage BiCross encoder for multilingual access to COVID-19 health information.
TL;DR: This paper proposed a novel high precision and high recall neural multi-stage BiCross encoder approach, which is a sequential three-stage ranking pipeline which uses the Okapi BM25 retrieval algorithm and transformer-based bi-encoder and crossencoder to effectively rank the documents with respect to the given query.