Open AccessProceedings Article
Attention is All you Need
Ashish Vaswani,Noam Shazeer,Niki Parmar,Jakob Uszkoreit,Llion Jones,Aidan N. Gomez,Lukasz Kaiser,Illia Polosukhin +7 more
- Vol. 30, pp 5998-6008
Reads0
Chats0
TLDR
This paper proposed a simple network architecture based solely on an attention mechanism, dispensing with recurrence and convolutions entirely and achieved state-of-the-art performance on English-to-French translation.Abstract:
The dominant sequence transduction models are based on complex recurrent orconvolutional neural networks in an encoder and decoder configuration. The best performing such models also connect the encoder and decoder through an attentionm echanisms. We propose a novel, simple network architecture based solely onan attention mechanism, dispensing with recurrence and convolutions entirely.Experiments on two machine translation tasks show these models to be superiorin quality while being more parallelizable and requiring significantly less timeto train. Our single model with 165 million parameters, achieves 27.5 BLEU onEnglish-to-German translation, improving over the existing best ensemble result by over 1 BLEU. On English-to-French translation, we outperform the previoussingle state-of-the-art with model by 0.7 BLEU, achieving a BLEU score of 41.1.read more
Citations
More filters
Proceedings ArticleDOI
Camouflaged Object Segmentation with Distraction Mining
TL;DR: Zhang et al. as mentioned in this paper developed a bio-inspired framework, termed Positioning and Focus Network (PFNet), which mimics the process of predation in nature, which contains two key modules, i.e., the positioning module (PM) and the focus module (FM).
Posted Content
Graph-Bert: Only Attention is Needed for Learning Graph Representations.
TL;DR: This paper introduces a new graph neural network, namely GRAPH-BERT (Graph based BERT), solely based on the attention mechanism without any graph convolution or aggregation operators, which can out-perform the existing GNNs in both the learning effectiveness and efficiency.
Proceedings ArticleDOI
ACFNet: Attentional Class Feature Network for Semantic Segmentation
TL;DR: ACFNet as mentioned in this paper proposes a coarse-to-fine segmentation network, which can be composed of an ACF module and any off-the-shell segmentation networks (base network).
Posted Content
Contrastive Bidirectional Transformer for Temporal Representation Learning
TL;DR: This paper adopts the stacked transformer architecture, but generalizes its training objective to maximize the mutual information between the masked signals, and the bidirectional context, via contrastive loss, which enables the model to handle continuous signals, such as visual features.
Posted Content
MISA: Modality-Invariant and -Specific Representations for Multimodal Sentiment Analysis
TL;DR: A novel framework, MISA, is proposed, which projects each modality to two distinct subspaces, which provide a holistic view of the multimodal data, which is used for fusion that leads to task predictions.