We introduce a new language representation model called BERT, which stands for Bidirectional Encoder Representations from Transformers. Unlike recent language representation models, BERT is designed to pre-train deep bidirectional representations from unlabeled text by jointly conditioning on both left and right context in all layers. As a result, the pre-trained BERT model can be fine-tuned with just one additional output layer to create state-of-the-art models for a wide range of tasks, such as question answering and language inference, without substantial task-specific architecture modifications. 
BERT is conceptually simple and empirically powerful. It obtains new state-of-the-art results on eleven natural language processing tasks, including pushing the GLUE score to 80.5% (7.7% point absolute improvement), MultiNLI accuracy to 86.7% (4.6% absolute improvement), SQuAD v1.1 question answering Test F1 to 93.2 (1.5 point absolute improvement) and SQuAD v2.0 Test F1 to 83.1 (5.1 point absolute improvement).

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

We introduce a new language representation model called BERT, which stands for Bidirectional Encoder Representations from Transformers. Unlike recent language representation models (Peters et al., 2018a; Radford et al., 2018), BERT is designed to pre-train deep bidirectional representations from unlabeled text by jointly conditioning on both left and right context in all layers. As a result, the pre-trained BERT model can be fine-tuned with just one additional output layer to create state-of-the-art models for a wide range of tasks, such as question answering and language inference, without substantial task-specific architecture modifications. BERT is conceptually simple and empirically powerful. It obtains new state-of-the-art results on eleven natural language processing tasks, including pushing the GLUE score to 80.5 (7.7 point absolute improvement), MultiNLI accuracy to 86.7% (4.6% absolute improvement), SQuAD v1.1 question answering Test F1 to 93.2 (1.5 point absolute improvement) and SQuAD v2.0 Test F1 to 83.1 (5.1 point absolute improvement).

Language model pretraining has led to significant performance gains but careful comparison between different approaches is challenging. Training is computationally expensive, often done on private datasets of different sizes, and, as we will show, hyperparameter choices have significant impact on the final results. We present a replication study of BERT pretraining (Devlin et al., 2019) that carefully measures the impact of many key hyperparameters and training data size. We find that BERT was significantly undertrained, and can match or exceed the performance of every model published after it. Our best model achieves state-of-the-art results on GLUE, RACE and SQuAD. These results highlight the importance of previously overlooked design choices, and raise questions about the source of recently reported improvements. We release our models and code.

/pdf/roberta-a-robustly-optimized-bert-pretraining-approach-2rj0bdyhim.pdf

RoBERTa: A Robustly Optimized BERT Pretraining Approach

A translation approach to portable ontology specifications

Recent work has demonstrated substantial gains on many NLP tasks and benchmarks by pre-training on a large corpus of text followed by fine-tuning on a specific task. While typically task-agnostic in architecture, this method still requires task-specific fine-tuning datasets of thousands or tens of thousands of examples. By contrast, humans can generally perform a new language task from only a few examples or from simple instructions - something which current NLP systems still largely struggle to do. Here we show that scaling up language models greatly improves task-agnostic, few-shot performance, sometimes even reaching competitiveness with prior state-of-the-art fine-tuning approaches. Specifically, we train GPT-3, an autoregressive language model with 175 billion parameters, 10x more than any previous non-sparse language model, and test its performance in the few-shot setting. For all tasks, GPT-3 is applied without any gradient updates or fine-tuning, with tasks and few-shot demonstrations specified purely via text interaction with the model. GPT-3 achieves strong performance on many NLP datasets, including translation, question-answering, and cloze tasks, as well as several tasks that require on-the-fly reasoning or domain adaptation, such as unscrambling words, using a novel word in a sentence, or performing 3-digit arithmetic. At the same time, we also identify some datasets where GPT-3's few-shot learning still struggles, as well as some datasets where GPT-3 faces methodological issues related to training on large web corpora. Finally, we find that GPT-3 can generate samples of news articles which human evaluators have difficulty distinguishing from articles written by humans. We discuss broader societal impacts of this finding and of GPT-3 in general.

/pdf/language-models-are-few-shot-learners-2fg8gvia7m.pdf

Language Models are Few-Shot Learners

In previous work, we proposed a modal fragment of the situation calculus called ${\mathcal ES}$, which fully captures Reiter's basic action theories. ${\mathcal ES}$ also has epistemic features, including only-knowing, which refers to all that an agent knows in the sense of having a knowledge base. While our model of only-knowing has appealing properties in the static case, it appears to be problematic when actions come into play. First of all, its utility seems to be restricted to an agent's initial knowledge base. Second, while it has been shown that only-knowing correctly captures default inferences, this was only in the static case, and undesirable properties appear to arise in the presence of actions. In this paper, we remedy both of these shortcomings and propose a new dynamic semantics of only-knowing, which is closely related to Lin and Reiter's notion of progression when actions are performed and where defaults behave properly.

/pdf/a-semantical-account-of-progression-in-the-presence-of-5f0anqz0im.pdf

A semantical account of progression in the presence of defaults

This work proposes a new methodology for establishing the tractability of a reasoning service that deals with expressive first-order knowledge bases. It consists of defining a logic that is weaker than classical logic and that has two properties: first, the entailment problem can be reduced to the model checking problem for a small number of characteristic models; and second, the model checking problem itself is tractable for formulas with a bounded number of variables. We show this methodology in action for the reasoning service previously proposed by Liu, Lakemeyer and Levesque for dealing with disjunctive information. They show that their reasoning is tractable in the propositional case and decidable in the first-order case. Here we apply the methodology and prove that the reasoning is also tractable in the first-order case if the knowledge base and the query both use a bounded number of variables.

/pdf/tractable-reasoning-in-first-order-knowledge-bases-with-2vkwknm26c.pdf

Tractable reasoning in first-order knowledge bases with disjunctive information

Reasoning about degrees of belief in uncertain dynamic worlds is fundamental to many applications, such as robotics and planning, where actions modify state properties and sensors provide measurements, both of which are prone to noise. With the exception of limited cases such as Gaussian processes over linear phenomena, belief state evolution can be complex and hard to reason with in a general way. This paper proposes a framework with new results that allows the reduction of subjective probabilities after sensing and acting, both in discrete and continuous domains, to questions about the initial state only. We build on an expressive probabilistic first-order logical account by Bacchus, Halpern and Levesque, resulting in a methodology that, in principle, can be coupled with a variety of existing inference solutions.

/pdf/reasoning-about-probabilities-in-dynamic-systems-using-goal-55skbdw8e7.pdf

Reasoning about probabilities in dynamic systems using goal regression

Specifying Communicative Multi-Agent Systems (Invited Paper)

Constraint satisfaction provides a uniform framework for addressing a variety of combinatorial problems. Propositional proof complexity is the study of the minimum lengths of proofs in propositional logic. We view constraint satisfaction problems (CSPs) as generalizing propositional satisfiability, and adapt ideas from resolution proof complexity to their study. We study a previously defined resolution-like proof system for CSPs, denoted here NG-RES, and a new system C-RES. NG-RES is very natural, but C-RES is more powerful, and is equivalent to propositional resolution for CSPs equivalent to SAT. 
We examine CSP algorithms that employ standard techniques from the literature, and show they are resolution-based. They implicitly construct NG-RES or C-RES proofs. We also give resolution-based algorithms for binary CSPs with non-trivial upper bounds. For domain size d, we construct C-RES proofs in time O([ d/2]n), and for domain size 3 we obtain O(1.38027n), which is close to the best algorithmic upper bound. 
For resolution-based algorithms, minimum resolution proof size is a lower bound on running time. We demonstrate there are many CSP instances for which the shortest C-RES proof is of exponential size: Any k-CNF formula which is hard for propositional resolution induces two families of CSP instances which are hard for C-RES, one with domain size k and binary constraints, and one with domain size two and k-ary constraints. 
Based on a known relationship between clause size and refutation length for propositional resolution, we reduce C-RES length to the “sub-critical expansion” of CSP instances. We use this to obtain C-RES lower bounds for random CSPs. Known exponential resolution lower bounds for random instances of k-SAT are seen to imply lower bounds for equivalent random CSPs. For a more general family of random CSPs, we obtain exponential C-RES lower bounds when constraints are “loose”. Our results generalize those implied via random k-SAT in two respects: they apply to binary CSPs, and they hold for constraints which are “tighter” than propositional clauses. 
We also demonstrate a width-length relationship for NG-RES. This implies that tree-NG-RES is quasi-automatizable, and may lead to an improved separation between NG-RES and RES.

Hector J. Levesque

Papers

A semantical account of progression in the presence of defaults

Tractable reasoning in first-order knowledge bases with disjunctive information

Reasoning about probabilities in dynamic systems using goal regression

Specifying Communicative Multi-Agent Systems (Invited Paper)

The resolution complexity of constraint satisfaction