We introduce a new language representation model called BERT, which stands for Bidirectional Encoder Representations from Transformers. Unlike recent language representation models, BERT is designed to pre-train deep bidirectional representations from unlabeled text by jointly conditioning on both left and right context in all layers. As a result, the pre-trained BERT model can be fine-tuned with just one additional output layer to create state-of-the-art models for a wide range of tasks, such as question answering and language inference, without substantial task-specific architecture modifications. 
BERT is conceptually simple and empirically powerful. It obtains new state-of-the-art results on eleven natural language processing tasks, including pushing the GLUE score to 80.5% (7.7% point absolute improvement), MultiNLI accuracy to 86.7% (4.6% absolute improvement), SQuAD v1.1 question answering Test F1 to 93.2 (1.5 point absolute improvement) and SQuAD v2.0 Test F1 to 83.1 (5.1 point absolute improvement).

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

We introduce a new language representation model called BERT, which stands for Bidirectional Encoder Representations from Transformers. Unlike recent language representation models (Peters et al., 2018a; Radford et al., 2018), BERT is designed to pre-train deep bidirectional representations from unlabeled text by jointly conditioning on both left and right context in all layers. As a result, the pre-trained BERT model can be fine-tuned with just one additional output layer to create state-of-the-art models for a wide range of tasks, such as question answering and language inference, without substantial task-specific architecture modifications. BERT is conceptually simple and empirically powerful. It obtains new state-of-the-art results on eleven natural language processing tasks, including pushing the GLUE score to 80.5 (7.7 point absolute improvement), MultiNLI accuracy to 86.7% (4.6% absolute improvement), SQuAD v1.1 question answering Test F1 to 93.2 (1.5 point absolute improvement) and SQuAD v2.0 Test F1 to 83.1 (5.1 point absolute improvement).

Language model pretraining has led to significant performance gains but careful comparison between different approaches is challenging. Training is computationally expensive, often done on private datasets of different sizes, and, as we will show, hyperparameter choices have significant impact on the final results. We present a replication study of BERT pretraining (Devlin et al., 2019) that carefully measures the impact of many key hyperparameters and training data size. We find that BERT was significantly undertrained, and can match or exceed the performance of every model published after it. Our best model achieves state-of-the-art results on GLUE, RACE and SQuAD. These results highlight the importance of previously overlooked design choices, and raise questions about the source of recently reported improvements. We release our models and code.

/pdf/roberta-a-robustly-optimized-bert-pretraining-approach-2rj0bdyhim.pdf

RoBERTa: A Robustly Optimized BERT Pretraining Approach

A translation approach to portable ontology specifications

Recent work has demonstrated substantial gains on many NLP tasks and benchmarks by pre-training on a large corpus of text followed by fine-tuning on a specific task. While typically task-agnostic in architecture, this method still requires task-specific fine-tuning datasets of thousands or tens of thousands of examples. By contrast, humans can generally perform a new language task from only a few examples or from simple instructions - something which current NLP systems still largely struggle to do. Here we show that scaling up language models greatly improves task-agnostic, few-shot performance, sometimes even reaching competitiveness with prior state-of-the-art fine-tuning approaches. Specifically, we train GPT-3, an autoregressive language model with 175 billion parameters, 10x more than any previous non-sparse language model, and test its performance in the few-shot setting. For all tasks, GPT-3 is applied without any gradient updates or fine-tuning, with tasks and few-shot demonstrations specified purely via text interaction with the model. GPT-3 achieves strong performance on many NLP datasets, including translation, question-answering, and cloze tasks, as well as several tasks that require on-the-fly reasoning or domain adaptation, such as unscrambling words, using a novel word in a sentence, or performing 3-digit arithmetic. At the same time, we also identify some datasets where GPT-3's few-shot learning still struggles, as well as some datasets where GPT-3 faces methodological issues related to training on large web corpora. Finally, we find that GPT-3 can generate samples of news articles which human evaluators have difficulty distinguishing from articles written by humans. We discuss broader societal impacts of this finding and of GPT-3 in general.

/pdf/language-models-are-few-shot-learners-2fg8gvia7m.pdf

Language Models are Few-Shot Learners

Reasoning about noisy sensors and effectors in the situation calculus

Brokered multi-agent systems can be incapacitated and rendered non-functional when the brokers become inaccessible due to failures that can occur in any distributed software system. We propose that the theory of teamwork can be used to specify robust brokered architectures that can recover from broker failures, and we present the adaptive agent architecture (AAA) to show the feasibility of this approach. The previous teamwork theory based on joint intentions assumes that team members remain in a team as long as the team exists. We extend this theory to allow dynamic broker teams whose members can change with time. We also introduce a theory of restorative maintenance goals that enables the brokers in an AAA broker team to start new brokers and recruit them to the broker team. As a result, an AAA-based multi-agent system can maintain a specified number of functional brokers in the system despite broker failures, thus effectively becoming a self-healing system.

The adaptive agent architecture: achieving fault-tolerance using persistent broker teams

This paper describes a novel approach to high-level agent programming based on a highly developed logical theory of action. The user provides a specification of the agents' basic actions (preconditions and effects) as well as of relevant aspects of the environment, in an extended version of the situation calculus. He can then specify behaviors for the agents in terms of these actions in a programming language where one can refer to conditions in effect in the environment. When an implementation of the basic actions is provided, the programs can be executed in a real environment; otherwise, a simulated execution is still possible. The interpreter automatically maintains the world model required to execute programs based on the specification. The theoretical framework includes a solution to the frame problem, allows agents to have incomplete knowledge of their environment, and handles perceptual actions. The theory can also be used to prove programs correct. A simple meeting scheduling application is used to present the approach. Ongoing work on implementing the approach and handling outstanding problems is also described.

/pdf/foundations-of-a-logical-approach-to-agent-programming-369b2opj15.pdf

Foundations of a logical approach to agent programming

An Overview of Knowledge Representation.

Like classical planning, the execution of high-level agent programs requires a reasoner to look all the way to a final goal state before even a single action can be taken in the world. This deferral is a serious problem in practice for large programs. Furthermore, the problem is compounded in the presence of sensing actions which provide necessary information, but only after they are executed in the world. To deal with this, we propose (characterize formally in the situation calculus, and implement in Prolog) a new incremental way of interpreting such high-level programs and a new high-level language construct, which together allow much more control to be exercised over when actions can be executed. We argue that such a scheme leads to a practical way to deal with large agent programs containing both nondeterminism and sensing.

/pdf/an-incremental-interpreter-for-high-level-programs-with-3099motinw.pdf

Hector J. Levesque

Papers

Reasoning about noisy sensors and effectors in the situation calculus

The adaptive agent architecture: achieving fault-tolerance using persistent broker teams

Foundations of a logical approach to agent programming

An Overview of Knowledge Representation.

An Incremental Interpreter for High-Level Programs with Sensing