We introduce a new language representation model called BERT, which stands for Bidirectional Encoder Representations from Transformers. Unlike recent language representation models, BERT is designed to pre-train deep bidirectional representations from unlabeled text by jointly conditioning on both left and right context in all layers. As a result, the pre-trained BERT model can be fine-tuned with just one additional output layer to create state-of-the-art models for a wide range of tasks, such as question answering and language inference, without substantial task-specific architecture modifications. 
BERT is conceptually simple and empirically powerful. It obtains new state-of-the-art results on eleven natural language processing tasks, including pushing the GLUE score to 80.5% (7.7% point absolute improvement), MultiNLI accuracy to 86.7% (4.6% absolute improvement), SQuAD v1.1 question answering Test F1 to 93.2 (1.5 point absolute improvement) and SQuAD v2.0 Test F1 to 83.1 (5.1 point absolute improvement).

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

We introduce a new language representation model called BERT, which stands for Bidirectional Encoder Representations from Transformers. Unlike recent language representation models (Peters et al., 2018a; Radford et al., 2018), BERT is designed to pre-train deep bidirectional representations from unlabeled text by jointly conditioning on both left and right context in all layers. As a result, the pre-trained BERT model can be fine-tuned with just one additional output layer to create state-of-the-art models for a wide range of tasks, such as question answering and language inference, without substantial task-specific architecture modifications. BERT is conceptually simple and empirically powerful. It obtains new state-of-the-art results on eleven natural language processing tasks, including pushing the GLUE score to 80.5 (7.7 point absolute improvement), MultiNLI accuracy to 86.7% (4.6% absolute improvement), SQuAD v1.1 question answering Test F1 to 93.2 (1.5 point absolute improvement) and SQuAD v2.0 Test F1 to 83.1 (5.1 point absolute improvement).

Language model pretraining has led to significant performance gains but careful comparison between different approaches is challenging. Training is computationally expensive, often done on private datasets of different sizes, and, as we will show, hyperparameter choices have significant impact on the final results. We present a replication study of BERT pretraining (Devlin et al., 2019) that carefully measures the impact of many key hyperparameters and training data size. We find that BERT was significantly undertrained, and can match or exceed the performance of every model published after it. Our best model achieves state-of-the-art results on GLUE, RACE and SQuAD. These results highlight the importance of previously overlooked design choices, and raise questions about the source of recently reported improvements. We release our models and code.

/pdf/roberta-a-robustly-optimized-bert-pretraining-approach-2rj0bdyhim.pdf

RoBERTa: A Robustly Optimized BERT Pretraining Approach

A translation approach to portable ontology specifications

Recent work has demonstrated substantial gains on many NLP tasks and benchmarks by pre-training on a large corpus of text followed by fine-tuning on a specific task. While typically task-agnostic in architecture, this method still requires task-specific fine-tuning datasets of thousands or tens of thousands of examples. By contrast, humans can generally perform a new language task from only a few examples or from simple instructions - something which current NLP systems still largely struggle to do. Here we show that scaling up language models greatly improves task-agnostic, few-shot performance, sometimes even reaching competitiveness with prior state-of-the-art fine-tuning approaches. Specifically, we train GPT-3, an autoregressive language model with 175 billion parameters, 10x more than any previous non-sparse language model, and test its performance in the few-shot setting. For all tasks, GPT-3 is applied without any gradient updates or fine-tuning, with tasks and few-shot demonstrations specified purely via text interaction with the model. GPT-3 achieves strong performance on many NLP datasets, including translation, question-answering, and cloze tasks, as well as several tasks that require on-the-fly reasoning or domain adaptation, such as unscrambling words, using a novel word in a sentence, or performing 3-digit arithmetic. At the same time, we also identify some datasets where GPT-3's few-shot learning still struggles, as well as some datasets where GPT-3 faces methodological issues related to training on large web corpora. Finally, we find that GPT-3 can generate samples of news articles which human evaluators have difficulty distinguishing from articles written by humans. We discuss broader societal impacts of this finding and of GPT-3 in general.

/pdf/language-models-are-few-shot-learners-2fg8gvia7m.pdf

Language Models are Few-Shot Learners

This is a brief overview of terminology and issues related to Knowledge Representation (here-after KR) research, intended primarily for researchers working on Semantic Data Models within Database Management and Program Specifications within Programming Languages/Software Engineering.
 Knowledge Representation is a central problem in Artificial Intelligence (AI) today. Its importance stems from the fact that the current design paradigm for “intelligent” systems stresses the need for expert knowledge in the system along with associated knowledge-handling facilities. This paradigm is in sharp contrast to earlier ones which might be termed “power-oriented” [Goldstein and Papert 77] in that they placed an emphasis on general purpose heuristic search techniques [Nilsson 71].

An Overview of Knowledge Representation

Publisher Summary Cognitive robotics is the study of the knowledge representation and reasoning problems faced by an autonomous robot (or an agent) in a dynamic and incompletely known world. This chapter presents the idea of knowledge representation and reasoning for the purpose of high-level robotic control to be central to cognitive robotics. This connects cognitive robotics not only to (traditional and less cognitive) robotics but also to other areas of artificial intelligence (AI) such as planning and agent-oriented programming. To illustrate the knowledge representation and reasoning issues relevant to high-level robotic control, the chapter discusses a Reiter's variant of the situation calculus. The situation calculus also deals with actions whose effects are deterministic—that is, where there is no doubt as to which ﬂuents change and which do not. The chapter discusses some of the knowledge representation issues that arise in the context of cognitive robotics and describes problems in automated reasoning in the same setting.

Chapter 23 Cognitive Robotics

This paper describes a novel approach to high-level robot programming based on a highly developed logica] theory of action. The user provides a specification of the robot’s basic actions (their preconditions and effects on the environment) as well as of relevant aspects of the environment, in an extended version of the situation calculus. He can then specify robot behaviors in terms of these actions in a programming language that allows references to world conditions (e.g. if 3c(PoP-CAN(c) A ON_TABLE(c)) then PICK_UP(c)). The programs can be executed to drive the robot. The interpreter automatically maintains the world model required to execute programs based on the specification. The theoretical framework includes a solution to the frame problem and is very general -- it handles dynamic and incompletely known worlds, as well as perception actions. Given this kind of domain specification, it is Mso possible to support more sophisticated reasoning, such as task planning at run-time. The specification can also be used to prove the robot control programs correct. A simple mall delivery application is used to present the approach. Ongoing work on implementing the approach and handling outstanding problems, such as modeling noisy sensors, is also described.

/pdf/a-logical-approach-to-high-level-robot-programming-a-h3xnve74hl.pdf

A Logical Approach to High-Level Robot Programming A Progress Report*

As an alternative to planning, an approach to highlevel agent control based on concurrent program execution is considered. A formal definition in the situation calculus of such a programming language is presented and illustrated with a detailed example. The language includes facilities for prioritizing the concurrent execution, interrupting the execution when certain conditions become true, and dealing with exogenous actions. The language differs from other procedural formalisms for concurrency in that the initial state can be incompletely specified and the primitive actions can be user-defined by axioms in the situation calculus.

/pdf/reasoning-about-concurrent-execution-prioritized-interrupts-533gj54joo.pdf

Reasoning about concurrent execution, prioritized interrupts, and exogenous actions in the situation calculus

We argue that current plan-based theories of discourse do not by themselves explain prevalent phenomena in even simple task-oriented dialogues. The purpose of this paper is to show how one difhcult-to-explain feature of these dialogues, confirmations, follows from the joint or team nature of the underlying task. Specifically, we review the concept of a joint intention and we argue that the conversants in a task-oriented dialogue jointly intend to accomplish the task. From this basis, we derive the goals underlying the pervasive use of confirmations observed in a recent experiment. We conclude with a discussion on generalizing the analysis presented here to characterize dialogue itself as a joint activity.

/pdf/confirmations-and-joint-action-2dxymou1v5.pdf

Hector J. Levesque

Papers

An Overview of Knowledge Representation

Chapter 23 Cognitive Robotics

A Logical Approach to High-Level Robot Programming A Progress Report*

Reasoning about concurrent execution, prioritized interrupts, and exogenous actions in the situation calculus

Confirmations and joint action