Open AccessProceedings Article
Attention is All you Need
Ashish Vaswani,Noam Shazeer,Niki Parmar,Jakob Uszkoreit,Llion Jones,Aidan N. Gomez,Lukasz Kaiser,Illia Polosukhin +7 more
- Vol. 30, pp 5998-6008
Reads0
Chats0
TLDR
This paper proposed a simple network architecture based solely on an attention mechanism, dispensing with recurrence and convolutions entirely and achieved state-of-the-art performance on English-to-French translation.Abstract:
The dominant sequence transduction models are based on complex recurrent orconvolutional neural networks in an encoder and decoder configuration. The best performing such models also connect the encoder and decoder through an attentionm echanisms. We propose a novel, simple network architecture based solely onan attention mechanism, dispensing with recurrence and convolutions entirely.Experiments on two machine translation tasks show these models to be superiorin quality while being more parallelizable and requiring significantly less timeto train. Our single model with 165 million parameters, achieves 27.5 BLEU onEnglish-to-German translation, improving over the existing best ensemble result by over 1 BLEU. On English-to-French translation, we outperform the previoussingle state-of-the-art with model by 0.7 BLEU, achieving a BLEU score of 41.1.read more
Citations
More filters
Posted Content
Text Summarization with Pretrained Encoders
Yang Liu,Mirella Lapata +1 more
TL;DR: This paper proposed a novel document-level encoder based on BERT which is able to express the semantics of a document and obtain representations for its sentences, and proposed a new fine-tuning schedule which adopts different optimizers for the encoder and the decoder as a means of alleviating the mismatch between the two.
Proceedings ArticleDOI
CodeBERT: A Pre-Trained Model for Programming and Natural Languages
Zhangyin Feng,Daya Guo,Duyu Tang,Nan Duan,Xiaocheng Feng,Ming Gong,Linjun Shou,Bing Qin,Ting Liu,Daxin Jiang,Ming Zhou +10 more
TL;DR: CodeBERT as mentioned in this paper is a pre-trained model for natural language code search and code documentation generation with a hybrid objective function that incorporates the pre-training task of replaced token detection, which is to detect plausible alternatives sampled from generators.
Proceedings ArticleDOI
HERO: Hierarchical Encoder for Video+Language Omni-representation Pre-training
TL;DR: HELP, a novel framework for large-scale video+language omni-representation learning that achieves new state of the art on multiple benchmarks over Text-based Video/Video-moment Retrieval, Video Question Answering (QA), Video-and-language Inference and Video Captioning tasks across different domains is presented.
Posted Content
Feature Denoising for Improving Adversarial Robustness
TL;DR: Feature denoising networks as mentioned in this paper uses non-local means or other filters to denoise the features of CNNs and achieve state-of-the-art performance in both white-box and black-box attacks.
Journal ArticleDOI
Multi-Scale Self-Guided Attention for Medical Image Segmentation
Ashish Sinha,Jose Dolz +1 more
TL;DR: Compared to other state-of-the-art segmentation networks, this model yields better segmentation performance, increasing the accuracy of the predictions while reducing the standard deviation, which demonstrates the efficiency of the approach to generate precise and reliable automatic segmentations of medical images.