Dmytro Okhonko

TL;DR: This paper proposes replacing the sinusoidal positional embedding for transformers with convolutionally learned input representations that provide subsequent transformer blocks with relative positional information needed for discovering long-range relationships between local concepts.

TL;DR: State-of-the-art RNN-based as well as Transformer-based models and open-source detailed training recipes are implemented and seamlessly integrated into S2T workflows for multi-task learning or transfer learning.

TL;DR: The casual masking object provides a type of hybrid of the more common causal and masked language models, by enabling full generative modeling while also providing bidirectional context when generating the masked spans.

TL;DR: Fairseq S2T as mentioned in this paper is a fairseq extension for speech-to-text (S2T) modeling tasks such as end-toend speech recognition and speech to-text translation.

TL;DR: The EfficientQA competition as mentioned in this paper focused on open-domain question answering (QA), where systems take natural language questions as input and return natural language answers, and the aim of the competition was to build systems that can predict correct answers while also satisfying strict on-disk memory budgets.