mT5: A Massively Multilingual Pre-trained Text-to-Text Transformer
Linting Xue,Noah Constant,Adam Roberts,Mihir Kale,Rami Al-Rfou,Aditya Siddhant,Aditya Barua,Colin Raffel +7 more
- pp 483-498
Reads0
Chats0
TLDR
This paper proposed a multilingual variant of T5, mT5, which was pre-trained on a new Common Crawl-based dataset covering 101 languages and achieved state-of-the-art performance on many multilingual benchmarks.Abstract:
The recent “Text-to-Text Transfer Transformer” (T5) leveraged a unified text-to-text format and scale to attain state-of-the-art results on a wide variety of English-language NLP tasks. In this paper, we introduce mT5, a multilingual variant of T5 that was pre-trained on a new Common Crawl-based dataset covering 101 languages. We detail the design and modified training of mT5 and demonstrate its state-of-the-art performance on many multilingual benchmarks. We also describe a simple technique to prevent “accidental translation” in the zero-shot setting, where a generative model chooses to (partially) translate its prediction into the wrong language. All of the code and model checkpoints used in this work are publicly available.read more
Citations
More filters
Journal Article
mSLAM: Massively multilingual joint pre-training for speech and text
Ankur Bapna,Colin Cherry,Yu Zhang,Ye Jia,Melvin George Johnson,Yong Cheng,Simran Khanuja,Jason Riesa,A-C. Conneau +8 more
TL;DR: mSLAM is evaluated on several downstream speech understanding tasks and finds that joint pre-training with text improves quality on speech translation, speech intent classification and speech languageID while being competitive on multilingual ASR, when compared against speech-only pre- training.
Proceedings ArticleDOI
Language Models are Multilingual Chain-of-Thought Reasoners
Haoyue Shi,Mirac M. Suzgun,Markus Freitag,Xuezhi Wang,Suraj Srivats,Soroush Vosoughi,Hyung Won Chung,Yi Tay,Sebastian Ruder,Denny Zhou,Dipanjan Das,Jason Loh Seong Wei +11 more
TL;DR: It is shown that the multilingual reasoning abilities of language models extend to other tasks such as commonsense reasoning and word-in-context semantic judgment, and that models have strikingly strong mult bilingual reasoning abilities, even in underrepresented languages such as Bengali and Swahili.
Proceedings ArticleDOI
ARBERT & MARBERT: Deep Bidirectional Transformers for Arabic
TL;DR: The authors introduced two powerful deep bidirectional transformer-based models, ARBERT and MARBERT, for multi-dialectal Arabic language understanding evaluation, which achieved state-of-the-art results across the majority of tasks (37 out of 48 classification tasks, on the 42 datasets).
Proceedings ArticleDOI
Designing Effective Sparse Expert Models
Barret Zoph,Irwan Bello,Sameer Kumar,Nan Du,Yanping Huang,Jeffrey Dean,Noam Shazeer,William Fedus +7 more
TL;DR: This paper proposed a stable and transferable Mixture-of-Experts (MoE-32B) model with 269B parameters, with a computational cost comparable to a 32B dense encoder-decoder Transformer.
Posted Content
The GEM Benchmark: Natural Language Generation, its Evaluation and Metrics.
Sebastian Gehrmann,Tosin P. Adewumi,Karmanya Aggarwal,Pawan Sasanka Ammanamanchi,Aremu Anuoluwapo,Antoine Bosselut,Khyathi Raghavi Chandu,Miruna Clinciu,Dipanjan Das,Kaustubh Dhole,Wanyu Du,Esin Durmus,Ondřej Dušek,Chris Chinenye Emezue,Varun Gangal,Cristina Garbacea,Tatsunori Hashimoto,Yufang Hou,Yacine Jernite,Harsh Jhamtani,Yangfeng Ji,Shailza Jolly,Mihir Kale,Dhruv Kumar,Faisal Ladhak,Aman Madaan,Mounica Maddela,Khyati Mahajan,Saad Mahamood,Bodhisattwa Prasad Majumder,Pedro Henrique Martins,Angelina McMillan-Major,Simon Mille,Emiel van Miltenburg,Moin Nadeem,Shashi Narayan,Vitaly Nikolaev,Rubungo Andre Niyongabo,Salomey Osei,Ankur P. Parikh,Laura Perez-Beltrachini,Niranjan Ramesh Rao,Vikas Raunak,Juan Diego Rodriguez,Sashank Santhanam,João Sedoc,Thibault Sellam,Samira Shaikh,Anastasia Shimorina,Marco Antonio Sobrevilla Cabezudo,Hendrik Strobelt,Nishant Subramani,Wei Xu,Diyi Yang,Akhila Yerukola,Jiawei Zhou +55 more
TL;DR: GEM as discussed by the authors is a living benchmark for natural language generation (NLG), its Evaluation and Metrics, which provides an environment in which models can easily be applied to a wide set of tasks and in which evaluation strategies can be tested.
References
More filters
Proceedings Article
Attention is All you Need
Ashish Vaswani,Noam Shazeer,Niki Parmar,Jakob Uszkoreit,Llion Jones,Aidan N. Gomez,Lukasz Kaiser,Illia Polosukhin +7 more
TL;DR: This paper proposed a simple network architecture based solely on an attention mechanism, dispensing with recurrence and convolutions entirely and achieved state-of-the-art performance on English-to-French translation.
Posted Content
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Yinhan Liu,Myle Ott,Naman Goyal,Jingfei Du,Mandar Joshi,Danqi Chen,Omer Levy,Michael Lewis,Luke Zettlemoyer,Veselin Stoyanov +9 more
TL;DR: It is found that BERT was significantly undertrained, and can match or exceed the performance of every model published after it, and the best model achieves state-of-the-art results on GLUE, RACE and SQuAD.
Proceedings ArticleDOI
SQuAD: 100,000+ Questions for Machine Comprehension of Text
TL;DR: The Stanford Question Answering Dataset (SQuAD) as mentioned in this paper is a reading comprehension dataset consisting of 100,000+ questions posed by crowdworkers on a set of Wikipedia articles, where the answer to each question is a segment of text from the corresponding reading passage.
Proceedings ArticleDOI
Unsupervised Cross-lingual Representation Learning at Scale
Alexis Conneau,Kartikay Khandelwal,Naman Goyal,Vishrav Chaudhary,Guillaume Wenzek,Francisco Guzmán,Edouard Grave,Myle Ott,Luke Zettlemoyer,Veselin Stoyanov +9 more
TL;DR: It is shown that pretraining multilingual language models at scale leads to significant performance gains for a wide range of cross-lingual transfer tasks, and the possibility of multilingual modeling without sacrificing per-language performance is shown for the first time.
Proceedings ArticleDOI
Universal Language Model Fine-tuning for Text Classification
Jeremy Howard,Sebastian Ruder +1 more
TL;DR: Universal Language Model Fine-tuning (ULMFiT) as mentioned in this paper is an effective transfer learning method that can be applied to any task in NLP, and introduces techniques that are key for finetuning a language model.