Open AccessProceedings Article
Attention is All you Need
Ashish Vaswani,Noam Shazeer,Niki Parmar,Jakob Uszkoreit,Llion Jones,Aidan N. Gomez,Lukasz Kaiser,Illia Polosukhin +7 more
- Vol. 30, pp 5998-6008
Reads0
Chats0
TLDR
This paper proposed a simple network architecture based solely on an attention mechanism, dispensing with recurrence and convolutions entirely and achieved state-of-the-art performance on English-to-French translation.Abstract:
The dominant sequence transduction models are based on complex recurrent orconvolutional neural networks in an encoder and decoder configuration. The best performing such models also connect the encoder and decoder through an attentionm echanisms. We propose a novel, simple network architecture based solely onan attention mechanism, dispensing with recurrence and convolutions entirely.Experiments on two machine translation tasks show these models to be superiorin quality while being more parallelizable and requiring significantly less timeto train. Our single model with 165 million parameters, achieves 27.5 BLEU onEnglish-to-German translation, improving over the existing best ensemble result by over 1 BLEU. On English-to-French translation, we outperform the previoussingle state-of-the-art with model by 0.7 BLEU, achieving a BLEU score of 41.1.read more
Citations
More filters
Proceedings ArticleDOI
An Analysis of Encoder Representations in Transformer-Based Machine Translation
TL;DR: This work investigates the information that is learned by the attention mechanism in Transformer models with different translation quality, and sheds light on the relative strengths and weaknesses of the various encoder representations.
Posted Content
Measuring the Effects of Data Parallelism on Neural Network Training
Christopher J. Shallue,Jaehoon Lee,Joseph M. Antognini,Jascha Sohl-Dickstein,Roy Frostig,George E. Dahl +5 more
TL;DR: This work experimentally characterize the effects of increasing the batch size on training time, as measured by the number of steps necessary to reach a goal out-of-sample error, and study how this relationship varies with the training algorithm, model, and data set, and finds extremely large variation between workloads.
Posted Content
Recurrent Independent Mechanisms
Anirudh Goyal,Alex Lamb,Jordan Hoffmann,Shagun Sodhani,Sergey Levine,Yoshua Bengio,Bernhard Schölkopf +6 more
TL;DR: Recurrent Independent Mechanisms is proposed, a new recurrent architecture in which multiple groups of recurrent cells operate with nearly independent transition dynamics, communicate only sparingly through the bottleneck of attention, and are only updated at time steps where they are most relevant.
Posted Content
Video Object Segmentation using Space-Time Memory Networks
TL;DR: This work proposes a novel solution for semi-supervised video object segmentation by leveraging memory networks and learning to read relevant information from all available sources to better handle the challenges such as appearance changes and occlussions.
Proceedings ArticleDOI
Make-A-Video: Text-to-Video Generation without Text-Video Data
Uriel Singer,Adam Polyak,Thomas Hayes,Xiaoyue Yin,Jie An,Songyang Zhang,Qiyuan Hu,Harry Yang,Oron Ashual,Oran Gafni,Devi Parikh,Sonal Gupta,Yaniv Taigman +12 more
TL;DR: Make-A-Video sets the new state-of-the-art in text-to-video generation, as determined by both qualitative and quantitative measures.