Open AccessProceedings Article
Neural Machine Translation by Jointly Learning to Align and Translate
TLDR
It is conjecture that the use of a fixed-length vector is a bottleneck in improving the performance of this basic encoder-decoder architecture, and it is proposed to extend this by allowing a model to automatically (soft-)search for parts of a source sentence that are relevant to predicting a target word, without having to form these parts as a hard segment explicitly.Abstract:
Neural machine translation is a recently proposed approach to machine translation. Unlike the traditional statistical machine translation, the neural machine translation aims at building a single neural network that can be jointly tuned to maximize the translation performance. The models proposed recently for neural machine translation often belong to a family of encoder-decoders and consists of an encoder that encodes a source sentence into a fixed-length vector from which a decoder generates a translation. In this paper, we conjecture that the use of a fixed-length vector is a bottleneck in improving the performance of this basic encoder-decoder architecture, and propose to extend this by allowing a model to automatically (soft-)search for parts of a source sentence that are relevant to predicting a target word, without having to form these parts as a hard segment explicitly. With this new approach, we achieve a translation performance comparable to the existing state-of-the-art phrase-based system on the task of English-to-French translation. Furthermore, qualitative analysis reveals that the (soft-)alignments found by the model agree well with our intuition.read more
Citations
More filters
Proceedings ArticleDOI
Simple Recurrent Units for Highly Parallelizable Recurrence.
TL;DR: The Simple Recurrent Unit (SRU) as mentioned in this paper is a light recurrent unit that balances model capacity and scalability, which is designed to provide expressive recurrence, enable highly parallelized implementation, and comes with careful initialization to facilitate training of deep models.
Proceedings Article
Measuring Compositional Generalization: A Comprehensive Method on Realistic Data
Daniel Keysers,Nathanael Schärli,Nathan Scales,Hylke Buisman,Daniel Furrer,Sergii Kashubin,Nikola Momchev,Danila Sinopalnikov,Lukasz Stafiniak,Tibor Tihon,Dmitry Tsarkov,Xiao Wang,Marc van Zee,Olivier Bousquet +13 more
TL;DR: A novel method to systematically construct compositional generalization benchmarks by maximizing compound divergence while guaranteeing a small atom divergence between train and test sets is introduced, and it is demonstrated how this method can be used to create new compositionality benchmarks on top of the existing SCAN dataset.
Proceedings ArticleDOI
Answer-focused and Position-aware Neural Question Generation.
TL;DR: The experimental results show that the proposed answer-focused and position-aware neural question generation model significantly improves the baseline and outperforms the state-of-the-art system.
Posted Content
Evaluating Prerequisite Qualities for Learning End-to-End Dialog Systems
Jesse Dodge,Andreea Gane,Xiang Zhang,Antoine Bordes,Sumit Chopra,Alexander H. Miller,Arthur Szlam,Jason Weston +7 more
TL;DR: The authors proposed a suite of new tasks of a much larger scale that attempt to bridge the gap between the two regimes by selecting the domain of movies and providing tasks that test the ability of models to answer factual questions, provide personalization, carry short conversations about the two, and finally to perform on natural dialogs from Reddit.
Posted Content
The FLoRes Evaluation Datasets for Low-Resource Machine Translation: Nepali-English and Sinhala-English.
Francisco Guzmán,Peng-Jen Chen,Myle Ott,Juan Pino,Guillaume Lample,Philipp Koehn,Vishrav Chaudhary,Marc'Aurelio Ranzato +7 more
TL;DR: This work introduces the FLORES evaluation datasets for Nepali–English and Sinhala– English, based on sentences translated from Wikipedia, and demonstrates that current state-of-the-art methods perform rather poorly on this benchmark, posing a challenge to the research community working on low-resource MT.
References
More filters
Journal ArticleDOI
Long short-term memory
TL;DR: A novel, efficient, gradient based method called long short-term memory (LSTM) is introduced, which can learn to bridge minimal time lags in excess of 1000 discrete-time steps by enforcing constant error flow through constant error carousels within special units.
Proceedings ArticleDOI
Learning Phrase Representations using RNN Encoder--Decoder for Statistical Machine Translation
Kyunghyun Cho,Bart van Merriënboer,Caglar Gulcehre,Dzmitry Bahdanau,Fethi Bougares,Holger Schwenk,Yoshua Bengio,Yoshua Bengio,Yoshua Bengio +8 more
TL;DR: In this paper, the encoder and decoder of the RNN Encoder-Decoder model are jointly trained to maximize the conditional probability of a target sequence given a source sequence.
Journal ArticleDOI
Learning long-term dependencies with gradient descent is difficult
TL;DR: This work shows why gradient based learning algorithms face an increasingly difficult problem as the duration of the dependencies to be captured increases, and exposes a trade-off between efficient learning by gradient descent and latching on information for long periods.
Journal ArticleDOI
Bidirectional recurrent neural networks
Mike Schuster,Kuldip K. Paliwal +1 more
TL;DR: It is shown how the proposed bidirectional structure can be easily modified to allow efficient estimation of the conditional posterior probability of complete symbol sequences without making any explicit assumption about the shape of the distribution.
Journal ArticleDOI
A neural probabilistic language model
TL;DR: The authors propose to learn a distributed representation for words which allows each training sentence to inform the model about an exponential number of semantically neighboring sentences, which can be expressed in terms of these representations.