Open AccessJournal Article
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Colin Raffel,Noam Shazeer,Adam Roberts,Katherine Lee,Sharan Narang,Michael Matena,Yanqi Zhou,Wei Li,Peter J. Liu +8 more
TLDR
This article introduced a unified framework that converts all text-based language problems into a text-to-text format and compared pre-training objectives, architectures, unlabeled data sets, transfer approaches, and other factors on dozens of language understanding tasks.Abstract:
Transfer learning, where a model is first pre-trained on a data-rich task before being fine-tuned on a downstream task, has emerged as a powerful technique in natural language processing (NLP). The effectiveness of transfer learning has given rise to a diversity of approaches, methodology, and practice. In this paper, we explore the landscape of transfer learning techniques for NLP by introducing a unified framework that converts all text-based language problems into a text-to-text format. Our systematic study compares pre-training objectives, architectures, unlabeled data sets, transfer approaches, and other factors on dozens of language understanding tasks. By combining the insights from our exploration with scale and our new ``Colossal Clean Crawled Corpus'', we achieve state-of-the-art results on many benchmarks covering summarization, question answering, text classification, and more. To facilitate future work on transfer learning for NLP, we release our data set, pre-trained models, and code.read more
Citations
More filters
Proceedings ArticleDOI
Multi-Task Retrieval for Knowledge-Intensive Tasks
Jean Maillard,Vladimir Karpukhin,Fabio Petroni,Wen-tau Yih,Barlas Oguz,Veselin Stoyanov,Gargi Ghosh +6 more
TL;DR: This paper proposed a multi-task trained neural retrieval model to retrieve relevant contexts from a large corpus for tasks such as open-domain question answering and fact checking, which outperforms traditional methods like tf-idf and BM25.
Posted Content
AWS CORD-19 Search: A neural search engine for COVID-19 literature
Parminder Bhatia,Lan Liu,Kristjan Arumae,Nima Pourdamghani,Suyog Deshpande,Ben Snively,Mona Mona,Colby Wise,George Price,Shyam Ramaswamy,Xiaofei Ma,Ramesh Nallapati,Zhiheng Huang,Bing Xiang,Taha A. Kass-Hout +14 more
TL;DR: AWS CORD-19 Search (ACS) is presented, a public, COVID-19 specific, neural search engine that is powered by several machine learning systems to support natural language based searches and is top performing across these systems yielding quality results.
Proceedings ArticleDOI
Domain-Specific Pretraining for Vertical Search: Case Study on Biomedical Literature
Yu Wang,Jinchao Li,Tristan Naumann,Chenyan Xiong,Hao Cheng,Robert Tinn,Cliff Wong,Naoto Usuyama,Richard Rogahn,Zhihong Shen,Yang Qin,Eric Horvitz,Paul N. Bennett,Jianfeng Gao,Hoifung Poon +14 more
TL;DR: In this paper, the authors propose a general approach for vertical search based on domain-specific pretraining and present a case study for the biomedical domain, which performs comparably or better than the best systems in the official TREC-COVID evaluation.
Proceedings ArticleDOI
Hierarchical Speaker-Aware Sequence-to-Sequence Model for Dialogue Summarization
TL;DR: The authors proposed a hierarchical transformer-based model for dialogue summarization, which encodes dialogues from words to utterances and distinguishes the relationships between speakers and their corresponding personal pronouns clearly.
Proceedings ArticleDOI
Of Non-Linearity and Commutativity in BERT
TL;DR: In this paper, the authors proposed a method to measure the degree of non-linearity of different elements of transformers and found that skip connections are an inefficient yet important architectural element and that they cannot simply be replaced by attention blocks without a degradation in performance.