We introduce a new language representation model called BERT, which stands for Bidirectional Encoder Representations from Transformers. Unlike recent language representation models, BERT is designed to pre-train deep bidirectional representations from unlabeled text by jointly conditioning on both left and right context in all layers. As a result, the pre-trained BERT model can be fine-tuned with just one additional output layer to create state-of-the-art models for a wide range of tasks, such as question answering and language inference, without substantial task-specific architecture modifications. 
BERT is conceptually simple and empirically powerful. It obtains new state-of-the-art results on eleven natural language processing tasks, including pushing the GLUE score to 80.5% (7.7% point absolute improvement), MultiNLI accuracy to 86.7% (4.6% absolute improvement), SQuAD v1.1 question answering Test F1 to 93.2 (1.5 point absolute improvement) and SQuAD v2.0 Test F1 to 83.1 (5.1 point absolute improvement).

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

We introduce a new language representation model called BERT, which stands for Bidirectional Encoder Representations from Transformers. Unlike recent language representation models (Peters et al., 2018a; Radford et al., 2018), BERT is designed to pre-train deep bidirectional representations from unlabeled text by jointly conditioning on both left and right context in all layers. As a result, the pre-trained BERT model can be fine-tuned with just one additional output layer to create state-of-the-art models for a wide range of tasks, such as question answering and language inference, without substantial task-specific architecture modifications. BERT is conceptually simple and empirically powerful. It obtains new state-of-the-art results on eleven natural language processing tasks, including pushing the GLUE score to 80.5 (7.7 point absolute improvement), MultiNLI accuracy to 86.7% (4.6% absolute improvement), SQuAD v1.1 question answering Test F1 to 93.2 (1.5 point absolute improvement) and SQuAD v2.0 Test F1 to 83.1 (5.1 point absolute improvement).

Language model pretraining has led to significant performance gains but careful comparison between different approaches is challenging. Training is computationally expensive, often done on private datasets of different sizes, and, as we will show, hyperparameter choices have significant impact on the final results. We present a replication study of BERT pretraining (Devlin et al., 2019) that carefully measures the impact of many key hyperparameters and training data size. We find that BERT was significantly undertrained, and can match or exceed the performance of every model published after it. Our best model achieves state-of-the-art results on GLUE, RACE and SQuAD. These results highlight the importance of previously overlooked design choices, and raise questions about the source of recently reported improvements. We release our models and code.

/pdf/roberta-a-robustly-optimized-bert-pretraining-approach-2rj0bdyhim.pdf

RoBERTa: A Robustly Optimized BERT Pretraining Approach

A translation approach to portable ontology specifications

Recent work has demonstrated substantial gains on many NLP tasks and benchmarks by pre-training on a large corpus of text followed by fine-tuning on a specific task. While typically task-agnostic in architecture, this method still requires task-specific fine-tuning datasets of thousands or tens of thousands of examples. By contrast, humans can generally perform a new language task from only a few examples or from simple instructions - something which current NLP systems still largely struggle to do. Here we show that scaling up language models greatly improves task-agnostic, few-shot performance, sometimes even reaching competitiveness with prior state-of-the-art fine-tuning approaches. Specifically, we train GPT-3, an autoregressive language model with 175 billion parameters, 10x more than any previous non-sparse language model, and test its performance in the few-shot setting. For all tasks, GPT-3 is applied without any gradient updates or fine-tuning, with tasks and few-shot demonstrations specified purely via text interaction with the model. GPT-3 achieves strong performance on many NLP datasets, including translation, question-answering, and cloze tasks, as well as several tasks that require on-the-fly reasoning or domain adaptation, such as unscrambling words, using a novel word in a sentence, or performing 3-digit arithmetic. At the same time, we also identify some datasets where GPT-3's few-shot learning still struggles, as well as some datasets where GPT-3 faces methodological issues related to training on large web corpora. Finally, we find that GPT-3 can generate samples of news articles which human evaluators have difficulty distinguishing from articles written by humans. We discuss broader societal impacts of this finding and of GPT-3 in general.

/pdf/language-models-are-few-shot-learners-2fg8gvia7m.pdf

Language Models are Few-Shot Learners

, In this paper, we describe a framework for specifying communicative multi-agent systems, using a theory of action based on the situation calculus to describe the effects of actions on the world and on the mental states of agents; and the concurrent, logic programming language ConGolog to specify the actions performed by each agent. Since ConGolog has a well-defined semantics, the specifications can be used to reason about the behavior of individual agents and the system as a whole. We extend the work presented in [7] to allow the specifications to mention agents' goals explicitly. The framework presented here allows the behavior of different agents to be specified at different levels of abstraction, using a rich set of programming language constructs. As an example, we specify a meeting scheduler multi-agent system.

Specifying communicative multi-agent systems

/pdf/logical-foundations-for-cognitive-agents-1e8k59nck1.pdf

Logical Foundations for Cognitive Agents

The relative expressive power of a sentential operator □α is compared to that of a syntactical predicate L(‘α’) in the setting of first-order logics. Despite well-known results by Montague and by Thomason that claim otherwise, any of the so-called “modal” logics of knowledge and belief can be compiled into classical first-order logics that have a corresponding predicate on sentences. Moreover, through the use of a partial truth predicate, the standard modal axiom schemata can be translated into single sentences, making it possible to use conventional first-order logic theorem provers to directly derive results in a wide class of modal logics.



Le pouvoir expressif relatif d'un operateur propositionnel □α est comparea celui d'un predicat syntaxique L(«α») dans l'etablissement de logiques de premier ordre. Malgre les resultats de Montague et Thomason qui indiquent le contraire, n'importe laquelle des soi-disant logiques modales de connaissances et de croyances peut ětre compilee dans les logiques classiques de premier ordre ayant un predicat correspondant sur les propositions. Qui plus est, par l'utilisation d'un predicat partiellement vrai, les schemas d'axiome modal standard peuvent ětre traduits en propositions simples, rendant possible l'utilisation de demonstrateurs de theoremes de logique de premier ordre traditionnels pour obtenir des resultats directs dans une vaste gamme de logiques modales.

/pdf/the-consistency-of-syntactical-treatments-of-knowledge-how-13ec7nlxr7.pdf

The consistency of syntactical treatments of knowledge: how to compile quantificational modal logics into classical FOL

A team of agents is jointly able to achieve a goal if despite any incomplete knowledge they may have about the world or each other, they still know enough to be able to get to a goal state. Unlike in the single-agent case, the mere existence of a working plan is not enough as there may be several incompatible working plans and the agents may not be able to choose a share that coordinates with those of the others. Some formalizations of joint ability ignore this issue of coordination within a coalition. Others, including those based on game theory, deal with coordination, but require a complete specification of what the agents believe. Such a complete specification is often not available. Here we present a new formalization of joint ability based on logical entailment in the situation calculus that avoids both of these pitfalls.

/pdf/a-logical-theory-of-coordination-and-joint-ability-4v25gfcn73.pdf

A logical theory of coordination and joint ability

There are a number of more or less orthogonal ways of looking at the kind of knowledge-base management system (KBMS) envisaged in this book. Nonetheless, two very distinct viewpoints seem to dominate. The first deals with what has been called the symbol level, and focuses on the data structures and algorithms needed to process and make available to a wide variety of users the large amounts of information present in (potentially distributed) KBs. Proposals here might involve using locking, parallelism, inheritance hierarchies, backward-chaining inference, database-style management of secondary storage and other mechanisms.3 This level deals with how a machine views and processes the information in a KB.

Hector J. Levesque

Papers

Specifying communicative multi-agent systems

Logical Foundations for Cognitive Agents

The consistency of syntactical treatments of knowledge: how to compile quantificational modal logics into classical FOL

A logical theory of coordination and joint ability

The knowledge level of KBMS