Open AccessJournal Article
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Colin Raffel,Noam Shazeer,Adam Roberts,Katherine Lee,Sharan Narang,Michael Matena,Yanqi Zhou,Wei Li,Peter J. Liu +8 more
Reads0
Chats0
TLDR
This article introduced a unified framework that converts all text-based language problems into a text-to-text format and compared pre-training objectives, architectures, unlabeled data sets, transfer approaches, and other factors on dozens of language understanding tasks.Abstract:
Transfer learning, where a model is first pre-trained on a data-rich task before being fine-tuned on a downstream task, has emerged as a powerful technique in natural language processing (NLP). The effectiveness of transfer learning has given rise to a diversity of approaches, methodology, and practice. In this paper, we explore the landscape of transfer learning techniques for NLP by introducing a unified framework that converts all text-based language problems into a text-to-text format. Our systematic study compares pre-training objectives, architectures, unlabeled data sets, transfer approaches, and other factors on dozens of language understanding tasks. By combining the insights from our exploration with scale and our new ``Colossal Clean Crawled Corpus'', we achieve state-of-the-art results on many benchmarks covering summarization, question answering, text classification, and more. To facilitate future work on transfer learning for NLP, we release our data set, pre-trained models, and code.read more
Citations
More filters
Journal ArticleDOI
BERT-JAM: Maximizing the utilization of BERT for neural machine translation
TL;DR: BERT-JAM is proposed to fill the research gap in studying how to maximize the utilization of BERT for NMT tasks with a three-phase optimization strategy that progressively unfreezes different components to overcome catastrophic forgetting during fine-tuning.
Posted Content
Weak-Attention Suppression For Transformer Based Speech Recognition
Yangyang Shi,Yongqiang Wang,Chunyang Wu,Christian Fuegen,Frank Zhang,Duc Le,Ching-Feng Yeh,Michael L. Seltzer +7 more
TL;DR: This paper proposes Weak-Attention Suppression (WAS), a method that dynamically induces sparsity in attention probabilities that leads to consistent Word Error Rate (WER) improvement over strong transformer baselines and indicates the importance of lookahead in attention-based ASR models.
Proceedings ArticleDOI
Prompting Contrastive Explanations for Commonsense Reasoning Tasks
Benchmarking Transformer-based Language Models for Arabic Sentiment and Sarcasm Detection
Ibrahim Abu Farha,Walid Magdy +1 more
TL;DR: This article evaluated the performance of transformer-based language models on Arabic sentiment and sarcasm detection and found that the models achieving the best performance are those that are trained on only Arabic data, including dialectal Arabic, and use a larger number of parameters, such as the recently released MARBERT.
Posted Content
Automated News Summarization Using Transformers
TL;DR: This article presented a comprehensive comparison of a few transformer architecture based pre-trained models for text summarization and human generated summaries for evaluating and comparing the summaries generated by machine learning models.