Open AccessJournal Article
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Colin Raffel,Noam Shazeer,Adam Roberts,Katherine Lee,Sharan Narang,Michael Matena,Yanqi Zhou,Wei Li,Peter J. Liu +8 more
TLDR
This article introduced a unified framework that converts all text-based language problems into a text-to-text format and compared pre-training objectives, architectures, unlabeled data sets, transfer approaches, and other factors on dozens of language understanding tasks.Abstract:
Transfer learning, where a model is first pre-trained on a data-rich task before being fine-tuned on a downstream task, has emerged as a powerful technique in natural language processing (NLP). The effectiveness of transfer learning has given rise to a diversity of approaches, methodology, and practice. In this paper, we explore the landscape of transfer learning techniques for NLP by introducing a unified framework that converts all text-based language problems into a text-to-text format. Our systematic study compares pre-training objectives, architectures, unlabeled data sets, transfer approaches, and other factors on dozens of language understanding tasks. By combining the insights from our exploration with scale and our new ``Colossal Clean Crawled Corpus'', we achieve state-of-the-art results on many benchmarks covering summarization, question answering, text classification, and more. To facilitate future work on transfer learning for NLP, we release our data set, pre-trained models, and code.read more
Citations
More filters
Proceedings ArticleDOI
ANA at SemEval-2020 Task 4: MUlti-task learNIng for cOmmonsense reasoNing (UNION)
TL;DR: In this paper, a unified end-to-end framework for commonsense reasoning was proposed to learn more dynamics under the scope of commonsense Reasoning, which can generate more meaningful explanations.
Posted Content
CoCo: Controllable Counterfactuals for Evaluating Dialogue State Trackers
Shiyang Li,Semih Yavuz,Kazuma Hashimoto,Jia Li,Tong Niu,Nazneen Fatema Rajani,Xifeng Yan,Yingbo Zhou,Caiming Xiong +8 more
TL;DR: This paper proposed controllable counterfactuals (CoCo) to bridge the gap and evaluate dialogue state tracking (DST) models on novel scenarios, i.e., would the system successfully tackle the request if the user responded differently but still consistently with the dialogue flow?
Proceedings ArticleDOI
Exploring Listwise Evidence Reasoning with T5 for Fact Verification
TL;DR: The authors leveraged pretrained sequence-to-sequence transformer models for sentence selection and label prediction, two key sub-tasks in fact verification, and achieved a FEVER score of 75.87% on the blind test set.
Posted Content
BENDR: using transformers and a contrastive self-supervised learning task to learn from massive amounts of EEG data
TL;DR: In this article, a pre-trained model is adapted for encephalography modeling using a self-supervised training objective to learn compressed representations of raw data signals, which can be fine-tuned to a variety of downstream BCI and EEG classification tasks, outperforming prior work in more task-specific (sleep stage classification).
Posted Content
Pruning the Index Contents for Memory Efficient Open-Domain QA.
TL;DR: In this paper, the authors present a pipeline that combines retriever, passage re-ranker, extractive reader, generative reader and a simple way to combine them to prune the contents of a massive index such that the open-domain QA system altogether with index, OS, and library components fits into 6GiB docker image.