Open AccessProceedings Article
Attention is All you Need
Ashish Vaswani,Noam Shazeer,Niki Parmar,Jakob Uszkoreit,Llion Jones,Aidan N. Gomez,Lukasz Kaiser,Illia Polosukhin +7 more
- Vol. 30, pp 5998-6008
Reads0
Chats0
TLDR
This paper proposed a simple network architecture based solely on an attention mechanism, dispensing with recurrence and convolutions entirely and achieved state-of-the-art performance on English-to-French translation.Abstract:
The dominant sequence transduction models are based on complex recurrent orconvolutional neural networks in an encoder and decoder configuration. The best performing such models also connect the encoder and decoder through an attentionm echanisms. We propose a novel, simple network architecture based solely onan attention mechanism, dispensing with recurrence and convolutions entirely.Experiments on two machine translation tasks show these models to be superiorin quality while being more parallelizable and requiring significantly less timeto train. Our single model with 165 million parameters, achieves 27.5 BLEU onEnglish-to-German translation, improving over the existing best ensemble result by over 1 BLEU. On English-to-French translation, we outperform the previoussingle state-of-the-art with model by 0.7 BLEU, achieving a BLEU score of 41.1.read more
Citations
More filters
Proceedings ArticleDOI
All in One Bad Weather Removal Using Architectural Search
TL;DR: This paper proposes a method that can handle multiple bad weather degradations: rain, fog, snow and adherent raindrops using a single network and designs a novel adversarial learning scheme that only backpropagates the loss of a degradation type to the respective task-specific encoder.
Posted Content
Attention Interpretability Across NLP Tasks.
TL;DR: This work attempts to fill the gap by giving a comprehensive explanation which justifies both kinds of observations (i.e., when is attention interpretable and when it is not) and reinforces the claim of interpretability of attention through manual evaluation.
Proceedings ArticleDOI
Adversarial Training for Weakly Supervised Event Detection
TL;DR: The experiments show that the candidate selection and adversarial training can cooperate together to obtain more diverse and accurate training data for ED, and significantly outperform the state-of-the-art methods in various weakly supervised scenarios.
Proceedings ArticleDOI
Graph-based Knowledge Tracing: Modeling Student Proficiency Using Graph Neural Network
TL;DR: Empirical validations indicated that the proposed GNN-based knowledge tracing method could potentially improve the prediction of student performance and demonstrated more interpretable predictions compared to those of the previous methods, without the requirement of any additional information.
Proceedings ArticleDOI
End-To-End Audio-Visual Speech Recognition with Conformers
TL;DR: In this article, a hybrid CTC/Attention model based on a ResNet-18 and Convolution-augmented transformer (Conformer) is proposed for sentence-level speech recognition.