Open AccessJournal Article
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Colin Raffel,Noam Shazeer,Adam Roberts,Katherine Lee,Sharan Narang,Michael Matena,Yanqi Zhou,Wei Li,Peter J. Liu +8 more
Reads0
Chats0
TLDR
This article introduced a unified framework that converts all text-based language problems into a text-to-text format and compared pre-training objectives, architectures, unlabeled data sets, transfer approaches, and other factors on dozens of language understanding tasks.Abstract:
Transfer learning, where a model is first pre-trained on a data-rich task before being fine-tuned on a downstream task, has emerged as a powerful technique in natural language processing (NLP). The effectiveness of transfer learning has given rise to a diversity of approaches, methodology, and practice. In this paper, we explore the landscape of transfer learning techniques for NLP by introducing a unified framework that converts all text-based language problems into a text-to-text format. Our systematic study compares pre-training objectives, architectures, unlabeled data sets, transfer approaches, and other factors on dozens of language understanding tasks. By combining the insights from our exploration with scale and our new ``Colossal Clean Crawled Corpus'', we achieve state-of-the-art results on many benchmarks covering summarization, question answering, text classification, and more. To facilitate future work on transfer learning for NLP, we release our data set, pre-trained models, and code.read more
Citations
More filters
Proceedings ArticleDOI
Differentiable Open-Ended Commonsense Reasoning
TL;DR: DrFact is proposed, an efficient Differentiable model for multi-hop Reasoning over knowledge Facts, which outperforms strong baseline methods by a large margin and is evaluated to evaluate OpenCSR methods.
Proceedings ArticleDOI
What Do Position Embeddings Learn? An Empirical Study of Pre-Trained Language Model Positional Encoding
Yu-An Wang,Yun-Nung Chen +1 more
TL;DR: This paper focuses on providing a new insight of pre-trained position embeddings through feature-level analysis and empirical experiments on most of iconic NLP tasks, which can guide the future work to choose the suitable positional encoding function for specific tasks given the application property.
Proceedings ArticleDOI
The GEM Benchmark: Natural Language Generation, its Evaluation and Metrics
Sebastian Gehrmann,Tosin P. Adewumi,Karmanya Aggarwal,Pawan Sasanka Ammanamanchi,Anuoluwapo Aremu,Antoine Bosselut,Khyathi Raghavi Chandu,Miruna-Adriana Clinciu,Dipanjan Das,Kaustubh Dhole,Wanyu Du,Esin Durmus,Ondřej Dušek,Chris Chinenye Emezue,Varun Gangal,Cristina Garbacea,Tatsunori Hashimoto,Yufang Hou,Yacine Jernite,Harsh Jhamtani,Yangfeng Ji,Shailza Jolly,Mihir Kale,Dhruv Kumar,Faisal Ladhak,Aman Madaan,Mounica Maddela,Khyati Mahajan,Saad Mahamood,Bodhisattwa Prasad Majumder,Pedro Henrique Martins,Angelina McMillan-Major,Simon Mille,Emiel van Miltenburg,Moin Nadeem,Shashi Narayan,Vitaly Nikolaev,Andre Niyongabo Rubungo,Salomey Osei,Ankur P. Parikh,Laura Perez-Beltrachini,Niranjan Ramesh Rao,Vikas Raunak,Juan Diego Rodriguez,Sashank Santhanam,João Sedoc,Thibault Sellam,Samira Shaikh,Anastasia Shimorina,Marco Antonio Sobrevilla Cabezudo,Hendrik Strobelt,Nishant Subramani,Wei Xu,Diyi Yang,Akhila Yerukola,Jiawei Zhou +55 more
TL;DR: GEM as discussed by the authors is a living benchmark for natural language generation (NLG), its Evaluation and Metrics, which provides an environment in which models can easily be applied to a wide set of tasks and in which evaluation strategies can be tested.
Posted Content
XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scale
Arun Babu,Changhan Wang,Andros Tjandra,Kushal Lakhotia,Qiantong Xu,Naman Goyal,Kritika Singh,Patrick von Platen,Yatharth Saraf,Juan Pino,Alexei Baevski,Alexis Conneau,Michael Auli +12 more
TL;DR: XLS-R as mentioned in this paper is a large-scale model for cross-lingual speech representation learning based on wav2vec 2.0, which is trained with up to 2B parameters on nearly half a million hours of publicly available speech audio in 128 languages, an order of magnitude more public data than the largest known prior work.
Journal ArticleDOI
A Novel Deep Learning-Based Multilevel Parallel Attention Neural (MPAN) Model for Multidomain Arabic Sentiment Analysis
TL;DR: This paper proposed a novel deep learning-based multilevel parallel attention neural (MPAN) model that uses a simple positioning binary embedding scheme (PBES) to simultaneously compute contextualized embeddings at the character, word, and sentence levels.