Open AccessProceedings Article
Attention is All you Need
Ashish Vaswani,Noam Shazeer,Niki Parmar,Jakob Uszkoreit,Llion Jones,Aidan N. Gomez,Lukasz Kaiser,Illia Polosukhin +7 more
- Vol. 30, pp 5998-6008
Reads0
Chats0
TLDR
This paper proposed a simple network architecture based solely on an attention mechanism, dispensing with recurrence and convolutions entirely and achieved state-of-the-art performance on English-to-French translation.Abstract:
The dominant sequence transduction models are based on complex recurrent orconvolutional neural networks in an encoder and decoder configuration. The best performing such models also connect the encoder and decoder through an attentionm echanisms. We propose a novel, simple network architecture based solely onan attention mechanism, dispensing with recurrence and convolutions entirely.Experiments on two machine translation tasks show these models to be superiorin quality while being more parallelizable and requiring significantly less timeto train. Our single model with 165 million parameters, achieves 27.5 BLEU onEnglish-to-German translation, improving over the existing best ensemble result by over 1 BLEU. On English-to-French translation, we outperform the previoussingle state-of-the-art with model by 0.7 BLEU, achieving a BLEU score of 41.1.read more
Citations
More filters
Proceedings ArticleDOI
How Much Knowledge Can You Pack Into the Parameters of a Language Model
TL;DR: The authors fine-tuned pre-trained models to answer questions without access to any external context or knowledge, which scales with model size and performs competitively with open-domain systems that explicitly retrieve answers from an external knowledge source when answering questions.
Posted Content
Q8BERT: Quantized 8Bit BERT
TL;DR: This work shows how to perform quantization-aware training during the fine-tuning phase of BERT in order to compress BERT by 4x with minimal accuracy loss and the produced quantized model can accelerate inference speed if it is optimized for 8bit Integer supporting hardware.
Journal ArticleDOI
JCS: An Explainable COVID-19 Diagnosis System by Joint Classification and Segmentation
TL;DR: A novel Joint Classification and Segmentation (JCS) system to perform real-time and explainable COVID- 19 chest CT diagnosis and extensive experiments demonstrate that the proposed JCS diagnosis system is very efficient for CO VID-19 classification and segmentation.
Proceedings ArticleDOI
Multi-Modal Fusion Transformer for End-to-End Autonomous Driving
TL;DR: TransFuser as discussed by the authors integrates image and LiDAR representations using attention and achieves state-of-the-art driving performance while reducing collisions by 76% compared to geometry-based fusion.
Posted Content
How to Fine-Tune BERT for Text Classification?
TL;DR: In this article, the authors conduct exhaustive experiments to investigate different fine-tuning methods of BERT (Bidirectional Encoder Representations from Transformers) on text classification task.