Open AccessJournal Article
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Colin Raffel,Noam Shazeer,Adam Roberts,Katherine Lee,Sharan Narang,Michael Matena,Yanqi Zhou,Wei Li,Peter J. Liu +8 more
Reads0
Chats0
TLDR
This article introduced a unified framework that converts all text-based language problems into a text-to-text format and compared pre-training objectives, architectures, unlabeled data sets, transfer approaches, and other factors on dozens of language understanding tasks.Abstract:
Transfer learning, where a model is first pre-trained on a data-rich task before being fine-tuned on a downstream task, has emerged as a powerful technique in natural language processing (NLP). The effectiveness of transfer learning has given rise to a diversity of approaches, methodology, and practice. In this paper, we explore the landscape of transfer learning techniques for NLP by introducing a unified framework that converts all text-based language problems into a text-to-text format. Our systematic study compares pre-training objectives, architectures, unlabeled data sets, transfer approaches, and other factors on dozens of language understanding tasks. By combining the insights from our exploration with scale and our new ``Colossal Clean Crawled Corpus'', we achieve state-of-the-art results on many benchmarks covering summarization, question answering, text classification, and more. To facilitate future work on transfer learning for NLP, we release our data set, pre-trained models, and code.read more
Citations
More filters
Proceedings ArticleDOI
When Do You Need Billions of Words of Pretraining Data
TL;DR: The authors showed that the ability to encode linguistic features is almost certainly necessary for language understanding, but it is likely that other, unidentified, forms of knowledge are the major drivers of recent improvements in language understanding among large pretrained models.
Journal ArticleDOI
Adapting Bidirectional Encoder Representations from Transformers (BERT) to Assess Clinical Semantic Textual Similarity: Algorithm Development and Validation Study.
Klaus Kades,Jan Sellner,Gregor Koehler,Peter M. Full,Peter M. Full,T. Y. Emmy Lai,T. Y. Emmy Lai,Jens Kleesiek,Klaus H. Maier-Hein +8 more
TL;DR: This article used BERT for assessing the semantic textual similarity of clinical text data and developed three different approaches where they (1) added additional, handcrafted sentence similarity features to the classifier token of BERT and combined the results with more features in multiple regression estimators, (2) incorporated a built-in ensembling method, M-Heads, into BERT by duplicating the regression head and applying an adapted training strategy to facilitate the focus of the heads on different input patterns of the medical sentences, and (3) developed a graph-based similarity approach for medications
Posted Content
IndoNLG: Benchmark and Resources for Evaluating Indonesian Natural Language Generation
Samuel Cahyawijaya,Genta Indra Winata,Bryan Wilie,Karissa Vincentio,Xiaohong Li,Adhiguna Kuncoro,Sebastian Ruder,Zhi Yuan Lim,Syafri Bahar,Masayu Leylia Khodra,Ayu Purwarianti,Pascale Fung +11 more
TL;DR: IndoNLG as discussed by the authors is the first NLG benchmark for the Indonesian language for natural language generation (NLG), which covers six tasks: summarization, question answering, open chitchat, as well as three different language pairs of machine translation tasks.
Proceedings ArticleDOI
An Enhanced Knowledge Injection Model for Commonsense Generation.
Zhihao Fan,Yeyun Gong,Zhongyu Wei,Siyuan Wang,Yameng Huang,Jian Jiao,Xuanjing Huang,Nan Duan,Ruofei Zhang +8 more
TL;DR: This work integrates two additional modules into the pretrained encoder-decoder model for prototype modeling to enhance the knowledge injection procedure and shows results that significantly improves the performance on all the metrics.
Posted Content
Covidex: Neural Ranking Models and Keyword Search Infrastructure for the COVID-19 Open Research Dataset
Edwin Zhang,Nikhil Gupta,Raphael Tang,Xiao Han,Ronak Pradeep,Kuang Lu,Yue Zhang,Rodrigo Nogueira,Kyunghyun Cho,Hui Fang,Jimmy Lin +10 more
TL;DR: Covidex as discussed by the authors is a search engine that exploits the latest neural ranking models to provide information access to the COVID-19 Open Research Dataset curated by the Allen Institute for AI.