Open AccessJournal Article
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Colin Raffel,Noam Shazeer,Adam Roberts,Katherine Lee,Sharan Narang,Michael Matena,Yanqi Zhou,Wei Li,Peter J. Liu +8 more
Reads0
Chats0
TLDR
This article introduced a unified framework that converts all text-based language problems into a text-to-text format and compared pre-training objectives, architectures, unlabeled data sets, transfer approaches, and other factors on dozens of language understanding tasks.Abstract:
Transfer learning, where a model is first pre-trained on a data-rich task before being fine-tuned on a downstream task, has emerged as a powerful technique in natural language processing (NLP). The effectiveness of transfer learning has given rise to a diversity of approaches, methodology, and practice. In this paper, we explore the landscape of transfer learning techniques for NLP by introducing a unified framework that converts all text-based language problems into a text-to-text format. Our systematic study compares pre-training objectives, architectures, unlabeled data sets, transfer approaches, and other factors on dozens of language understanding tasks. By combining the insights from our exploration with scale and our new ``Colossal Clean Crawled Corpus'', we achieve state-of-the-art results on many benchmarks covering summarization, question answering, text classification, and more. To facilitate future work on transfer learning for NLP, we release our data set, pre-trained models, and code.read more
Citations
More filters
Journal ArticleDOI
GenNI: Human-AI Collaboration for Data-Backed Text Generation.
Hendrik Strobelt,Jambay Kinley,Robert Krueger,Johanna Beyer,Hanspeter Pfister,Alexander M. Rush +5 more
TL;DR: GenNI as discussed by the authors is an interactive visual system for high-level human-AI collaboration in producing descriptive text, which allows users to globally constrain model generations, without sacrificing the representation power of the deep learning models.
Proceedings ArticleDOI
Improving Commonsense Question Answering by Graph-based Iterative Retrieval over Multiple Knowledge Sources
TL;DR: A novel question-answering method by integrating multiple knowledge sources, i.e. ConceptNet, Wikipedia, and the Cambridge Dictionary, to boost the performance and introduces a novel graph-based iterative knowledge retrieval module, which iteratively retrieves concepts and entities related to the given question and its choices from multipleknowledge sources.
Proceedings ArticleDOI
Neural Network Surgery: Injecting Data Patterns into Pre-trained Models with Minimal Instance-wise Side Effects
TL;DR: It is demonstrated that side effects can be controlled by the number of changed parameters and thus, it is proposed to conduct neural network surgery by only modifying a limited number of parameters.
Proceedings ArticleDOI
Larger-Scale Transformers for Multilingual Masked Language Modeling
TL;DR: This article presented the results of two larger multilingual masked language models, with 3.5B and 10.7B parameters, which outperformed XLM-R by 1.8% and 2.4% average accuracy on XNLI.
Proceedings ArticleDOI
Predicting Clickbait Strength in Online Social Media
TL;DR: This work rigorously investigate the use of transformers for clickbait strength prediction and achieves superior results on a benchmark dataset with ∼39K posts, outperform all the existing methods in the Clickbait Challenge.