We introduce a new language representation model called BERT, which stands for Bidirectional Encoder Representations from Transformers. Unlike recent language representation models, BERT is designed to pre-train deep bidirectional representations from unlabeled text by jointly conditioning on both left and right context in all layers. As a result, the pre-trained BERT model can be fine-tuned with just one additional output layer to create state-of-the-art models for a wide range of tasks, such as question answering and language inference, without substantial task-specific architecture modifications. 
BERT is conceptually simple and empirically powerful. It obtains new state-of-the-art results on eleven natural language processing tasks, including pushing the GLUE score to 80.5% (7.7% point absolute improvement), MultiNLI accuracy to 86.7% (4.6% absolute improvement), SQuAD v1.1 question answering Test F1 to 93.2 (1.5 point absolute improvement) and SQuAD v2.0 Test F1 to 83.1 (5.1 point absolute improvement).

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

We introduce a new language representation model called BERT, which stands for Bidirectional Encoder Representations from Transformers. Unlike recent language representation models (Peters et al., 2018a; Radford et al., 2018), BERT is designed to pre-train deep bidirectional representations from unlabeled text by jointly conditioning on both left and right context in all layers. As a result, the pre-trained BERT model can be fine-tuned with just one additional output layer to create state-of-the-art models for a wide range of tasks, such as question answering and language inference, without substantial task-specific architecture modifications. BERT is conceptually simple and empirically powerful. It obtains new state-of-the-art results on eleven natural language processing tasks, including pushing the GLUE score to 80.5 (7.7 point absolute improvement), MultiNLI accuracy to 86.7% (4.6% absolute improvement), SQuAD v1.1 question answering Test F1 to 93.2 (1.5 point absolute improvement) and SQuAD v2.0 Test F1 to 83.1 (5.1 point absolute improvement).

Language model pretraining has led to significant performance gains but careful comparison between different approaches is challenging. Training is computationally expensive, often done on private datasets of different sizes, and, as we will show, hyperparameter choices have significant impact on the final results. We present a replication study of BERT pretraining (Devlin et al., 2019) that carefully measures the impact of many key hyperparameters and training data size. We find that BERT was significantly undertrained, and can match or exceed the performance of every model published after it. Our best model achieves state-of-the-art results on GLUE, RACE and SQuAD. These results highlight the importance of previously overlooked design choices, and raise questions about the source of recently reported improvements. We release our models and code.

/pdf/roberta-a-robustly-optimized-bert-pretraining-approach-2rj0bdyhim.pdf

RoBERTa: A Robustly Optimized BERT Pretraining Approach

A translation approach to portable ontology specifications

Recent work has demonstrated substantial gains on many NLP tasks and benchmarks by pre-training on a large corpus of text followed by fine-tuning on a specific task. While typically task-agnostic in architecture, this method still requires task-specific fine-tuning datasets of thousands or tens of thousands of examples. By contrast, humans can generally perform a new language task from only a few examples or from simple instructions - something which current NLP systems still largely struggle to do. Here we show that scaling up language models greatly improves task-agnostic, few-shot performance, sometimes even reaching competitiveness with prior state-of-the-art fine-tuning approaches. Specifically, we train GPT-3, an autoregressive language model with 175 billion parameters, 10x more than any previous non-sparse language model, and test its performance in the few-shot setting. For all tasks, GPT-3 is applied without any gradient updates or fine-tuning, with tasks and few-shot demonstrations specified purely via text interaction with the model. GPT-3 achieves strong performance on many NLP datasets, including translation, question-answering, and cloze tasks, as well as several tasks that require on-the-fly reasoning or domain adaptation, such as unscrambling words, using a novel word in a sentence, or performing 3-digit arithmetic. At the same time, we also identify some datasets where GPT-3's few-shot learning still struggles, as well as some datasets where GPT-3 faces methodological issues related to training on large web corpora. Finally, we find that GPT-3 can generate samples of news articles which human evaluators have difficulty distinguishing from articles written by humans. We discuss broader societal impacts of this finding and of GPT-3 in general.

/pdf/language-models-are-few-shot-learners-2fg8gvia7m.pdf

Language Models are Few-Shot Learners

A central problem in applying logical knowledge representation formalisms to traditional robotics is that the treatment of belief change is categorical in the former, while probabilistic in the latter. A typical example is the fundamental capability of localization where a robot uses its noisy sensors to situate itself in a dynamic world. Domain designers are then left with the rather unfortunate task of abstracting probabilistic sensors in terms of categorical ones, or more drastically, completely abandoning the inner workings of sensors to black-box probabilistic tools and then interpreting their outputs in an abstract way. Building on a first-principles approach by Bacchus, Halpern and Levesque, and a recent continuous extension to it by Belle and Levesque, we provide an axiomatization that shows how localization can be realized wrt a basic action theory, thereby demonstrating how such capabilities can be enabled in a single logical framework. We then show how the framework can also enable localization for multiple agents, where an agent can appeal to the sensing already performed by another agent and the knowledge of their relative positions to localize itself.

/pdf/a-logical-theory-of-localization-2d440hro64.pdf

A Logical Theory of Localization

There are not very many existing logics of belief which have both a perspicuous semantics and are computationally attractive. An exception is the logic SL, proposed by Liu, Lakemeyer, and Levesque, which allows for a decidable and often even tractable form of reasoning. While the language is first-order and hence quite expressive, it still has a number of shortcomings. For one, beliefs about beliefs are not addressed at all. For another, the names of individuals are rigid, that is, their identity is assumed to be known. In this paper, we show how both shortcomings can be overcome by suitably extending the language and its semantics. Among other things, we show that determining the beliefs of a certain kind of fully introspective knowledge bases is decidable and that unknown individuals in the knowledge base can be accommodated in a decidable manner as well.

Decidable reasoning in a logic of limited belief with introspection and unknown individuals

Most existing work in agent programming assumes an execution model where an agent has a knowledge base (KB) about the current state of the world, and makes decisions about what to do in terms of what is entailed or consistent with this KB. Planning then involves looking ahead and gauging what would be consistent or entailed at various stages by possible future KBs. We show that in the presence of sensing, such a model does not always work properly, and propose an alternative that does. We then discuss how this affects agent programming language design/semantics.

/pdf/on-ability-to-autonomously-execute-agent-programs-with-369loprl9g.pdf

On Ability to Autonomously Execute Agent Programs with Sensing

There has been much work already on formal models of deliberation/planning under incomplete information, where an agent can perform sensing actions to acquire additional information. But most of it has been set in epistemic logicbased frameworks and is hard to relate to work on agent programming languages (e.g. 3APL, AgentSpeak(L)). Here, we develop new non-epistemic formalizations of deliberation that are much easier to relate to standard agent programming language semantics based on transition systems. When doing deliberation/planning under incomplete information, one typically searches over a set of states, each of which is associated with a knowledge base (KB) or theory that represents what is known in the state. To evaluate tests in the program and to determine what transitions/actions are possible, one looks at what is entailed by the current KB. To allow for future sensing results, one looks at which of these are consistent with the current KB. We call this type of approach to deliberation “entailment and consistencybased” (EC-based). In this paper, we argue that EC-based approaches do not always work, and propose an alternative. Our accounts are formalized within the situation calculus and use a simple programming language based on ConGolog to specify agent programs, but we claim that the results generalize to most proposed agent programming languages/frameworks. Our accounts rely on a semantics for online executions of programs with sensing. A configuration is a pair (δ, σ) involving a program δ and a history σ specifying the actions performed so far and the sensing results obtained. In the full paper, we define a transition relation for this, i.e. when a configuration (δ, σ) may evolve to configuration (δ ,σ ) w.r.t. a model M (relative to an underlying theory of action D). The definition requires that the theory D, augmented with the sensing results in σ, entail that the transition is possible. The model M is used to represent a possible environment and generate sensing results. We also define when a configuration is final, i.e. may legally terminate.

https://www.dis.uniroma1.it/~degiacom/papers/2004/aamas04short.pdf

On Ability to Autonomously Execute Agent Programs with Sensing — Extended Abstract

We introduce three-rained extensions of major nonmonotonic formalisms and we prove that the recently proposed well-founded semantics of logic programs is equivalent, for arbitrary logic programs, to three-valued forms of McCarthy's circumscription, Reiter's closed world assumption, Moore's autoepistemic logic and Reiter's default theory. This result not only provides a further justification of the well-founded semantics as a natural extension of the perfect model semantics from the class of stratified programs to the class of all logic programs, but it also establishes the class of all logic programs as a large class of theories, for which natural forms of all four nonmonotonic formalisms coincide. lt also paves the way for using efficient computation methods, developed for logic programming, as inference mechanisms for nonmonotonic reasoning

Hector J. Levesque

Papers

A Logical Theory of Localization

Decidable reasoning in a logic of limited belief with introspection and unknown individuals

On Ability to Autonomously Execute Agent Programs with Sensing

On Ability to Autonomously Execute Agent Programs with Sensing — Extended Abstract

Three-valued nonmonotonic formalisms and semantics of logic programs