Doug Downey

TL;DR: It is consistently found that multi-phase adaptive pretraining offers large gains in task performance, and it is shown that adapting to a task corpus augmented using simple data selection strategies is an effective alternative, especially when resources for domain-adaptive pretraining might be unavailable.

TL;DR: This study introduces a challenge dataset, ART, that consists of over 20k commonsense narrative contexts and 200k explanations, and conceptualizes two new tasks -- Abductive NLI: a multiple-choice question answering task for choosing the more likely explanation, and Abduction NLG: a conditional generation task for explaining given observations in natural language.

TL;DR: The authors show that adapting to a task corpus augmented using simple data selection strategies is an effective alternative, especially when resources for domain-adaptive pretraining might be unavailable, and consistently find that multi-phase adaptive pretraining offers large gains in task performance.

TL;DR: For example, the authors investigate the feasibility of abductive reasoning in natural language inference and generation and show that the best model achieves 68.9% accuracy, well below human performance of 91.4%.

TL;DR: Specter as discussed by the authors proposes a new method to generate document-level embedding of scientific papers based on pretraining a Transformer language model on a powerful signal of documentlevel relatedness: the citation graph.