We introduce a new language representation model called BERT, which stands for Bidirectional Encoder Representations from Transformers. Unlike recent language representation models, BERT is designed to pre-train deep bidirectional representations from unlabeled text by jointly conditioning on both left and right context in all layers. As a result, the pre-trained BERT model can be fine-tuned with just one additional output layer to create state-of-the-art models for a wide range of tasks, such as question answering and language inference, without substantial task-specific architecture modifications. 
BERT is conceptually simple and empirically powerful. It obtains new state-of-the-art results on eleven natural language processing tasks, including pushing the GLUE score to 80.5% (7.7% point absolute improvement), MultiNLI accuracy to 86.7% (4.6% absolute improvement), SQuAD v1.1 question answering Test F1 to 93.2 (1.5 point absolute improvement) and SQuAD v2.0 Test F1 to 83.1 (5.1 point absolute improvement).

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

We introduce a new language representation model called BERT, which stands for Bidirectional Encoder Representations from Transformers. Unlike recent language representation models (Peters et al., 2018a; Radford et al., 2018), BERT is designed to pre-train deep bidirectional representations from unlabeled text by jointly conditioning on both left and right context in all layers. As a result, the pre-trained BERT model can be fine-tuned with just one additional output layer to create state-of-the-art models for a wide range of tasks, such as question answering and language inference, without substantial task-specific architecture modifications. BERT is conceptually simple and empirically powerful. It obtains new state-of-the-art results on eleven natural language processing tasks, including pushing the GLUE score to 80.5 (7.7 point absolute improvement), MultiNLI accuracy to 86.7% (4.6% absolute improvement), SQuAD v1.1 question answering Test F1 to 93.2 (1.5 point absolute improvement) and SQuAD v2.0 Test F1 to 83.1 (5.1 point absolute improvement).

Language model pretraining has led to significant performance gains but careful comparison between different approaches is challenging. Training is computationally expensive, often done on private datasets of different sizes, and, as we will show, hyperparameter choices have significant impact on the final results. We present a replication study of BERT pretraining (Devlin et al., 2019) that carefully measures the impact of many key hyperparameters and training data size. We find that BERT was significantly undertrained, and can match or exceed the performance of every model published after it. Our best model achieves state-of-the-art results on GLUE, RACE and SQuAD. These results highlight the importance of previously overlooked design choices, and raise questions about the source of recently reported improvements. We release our models and code.

/pdf/roberta-a-robustly-optimized-bert-pretraining-approach-2rj0bdyhim.pdf

RoBERTa: A Robustly Optimized BERT Pretraining Approach

A translation approach to portable ontology specifications

Recent work has demonstrated substantial gains on many NLP tasks and benchmarks by pre-training on a large corpus of text followed by fine-tuning on a specific task. While typically task-agnostic in architecture, this method still requires task-specific fine-tuning datasets of thousands or tens of thousands of examples. By contrast, humans can generally perform a new language task from only a few examples or from simple instructions - something which current NLP systems still largely struggle to do. Here we show that scaling up language models greatly improves task-agnostic, few-shot performance, sometimes even reaching competitiveness with prior state-of-the-art fine-tuning approaches. Specifically, we train GPT-3, an autoregressive language model with 175 billion parameters, 10x more than any previous non-sparse language model, and test its performance in the few-shot setting. For all tasks, GPT-3 is applied without any gradient updates or fine-tuning, with tasks and few-shot demonstrations specified purely via text interaction with the model. GPT-3 achieves strong performance on many NLP datasets, including translation, question-answering, and cloze tasks, as well as several tasks that require on-the-fly reasoning or domain adaptation, such as unscrambling words, using a novel word in a sentence, or performing 3-digit arithmetic. At the same time, we also identify some datasets where GPT-3's few-shot learning still struggles, as well as some datasets where GPT-3 faces methodological issues related to training on large web corpora. Finally, we find that GPT-3 can generate samples of news articles which human evaluators have difficulty distinguishing from articles written by humans. We discuss broader societal impacts of this finding and of GPT-3 in general.

/pdf/language-models-are-few-shot-learners-2fg8gvia7m.pdf

Language Models are Few-Shot Learners

Researchers in artificial intelligence have recent( been taking great interest in hybrid representations, among them sorted logics—logics that link a traditional logical representation to a taxonomie (or sort) representation such as those prevalent in semantic networks. This paper introduces a general framework—the substitutional framework—for integrating logical deduction and sortal deduction to form a deductive system for sorted logic. This paper also presents results that provide the theoretical underpinnings of the framework A distinguishing characteristic of a deductive system that is structured according to the substitutional framework is that the sort subsystem is invoked only when the logic subsystem performs unification, and thus sort information is used only in determining what substitutions to make for variables. Unlike every other known approach to sorted deduction, the substitutional framework provides for a systematic transformation of unsorted deductive systems into sorted ones

The substitutional framework for sorted deduction: fundamental results on hybrid reasoning

This chapter examines a new formalism—production systems—that is used extensively in practical applications and emphasizes forward chaining over rules as a way of reasoning. It also looks the examples where the reasoning is data-directed, and others where it is goal-directed. Applications built using production systems are often called rule-based systems as a way of highlighting the emphasis on rules in the underlying knowledge representation. Data-directed reasoning might be most appropriate in a database-like setting, when assertions are made and it is important to follow the implications of those assertions. Goal-directed reasoning might be most appropriate in a problem-solving situation, where a desired result is clear and the means to achieve that result—the logical foundations for a conclusion—are sought. The chapter also mentions the production rules as the antecedent of a production rule is a set of conditions. If there is more than one condition, they are understood conjunctively that they all are to be true for the rule to be applicable.

Rules in Production Systems

A central problem in applying logical knowledge representation formalisms to traditional robotics is that the treatment of belief change is categorical in the former, while probabilistic in the latter. A typical example is the fundamental capability of localization where a robot uses its many noisy sensors to situate itself in a dynamic world. Domain designers are then left with the rather unfortunate task of abstracting probabilistic sensors in terms of categorical ones, or more drastically, completely abandoning the inner workings of sensors to black-box probabilistic tools and then interpreting their outputs in an abstract way. Building on a first-principles approach by Bacchus, Halpern and Levesque, and a recent continuous extension to it by Belle and Levesque, we provide an axiomatization that shows how localization can be realized as a basic action theory, thereby demonstrating how such capabilities can be enabled in a single logical framework.

A Logical Theory of Robot Localization.

Levesque's proper knowledge bases proper KBs correspond to infinite sets of ground positive and negative facts, with the notable property that for FOL formulas in a certain normal form, which includes conjunctive queries and positive queries possibly extended with a controlled form of negation, entailment reduces to formula evaluation. However proper KBs represent extensional knowledge only. In description logic terms, they correspond to ABoxes. In this paper, we augment them with DL-Lite TBoxes, expressing intensional knowledge i.e., the ontology of the domain. DL-Lite has the notable property that conjunctive query answering over TBoxes and standard description logic ABoxes is reducible to formula evaluation over the ABox only. Here, we investigate whether such a property extends to ABoxes consisting of proper KBs. Specifically, we consider two DL-Lite variants: $$\textit{DL-Lite}_{ rdfs }$$, roughly corresponding to RDFS, and $$\textit{DL-Lite}_{ core }$$, roughly corresponding to OWL 2 QL. We show that when a $$\textit{DL-Lite}_{ rdfs }$$ TBox is coupled with a proper KB, the TBox can be compiled away, reducing query answering to evaluation on the proper KB alone. But this reduction is no longer possible when we associate proper KBs with $$\textit{DL-Lite}_{ core }$$ TBoxes. Indeed, we show that in the latter case, query answering even for conjunctive queries becomes coNP-hard in data complexity.

/pdf/adding-dl-lite-tboxes-to-proper-knowledge-bases-1f1kty99p8.pdf

Adding DL-Lite TBoxes to Proper Knowledge Bases

This paper describes an approach for reasoning in a dynamic domain with nondeterministic actions in which an agent’s (categorical) beliefs correspond to the simplest, or most plausible, course of events consistent with the agent’s observations and beliefs. The account is based on an epistemic extension of the situation calculus, a first-order theory of reasoning about action that accommodates sensing actions. In particular, the account is based on a qualitative theory of nondeterminism. Our position is that for commonsense reasoning, the world is most usefully regarded as deterministic, and that nondeterminism is an epistemic phenomenon, arising from an agent’s limited awareness and perception. The account offers several advantages: an agent has a set of categorical (as opposed to probabilistic) beliefs, yet can deal with equally-likely outcomes (such as in flipping a fair coin) or with outcomes of differing plausibility (such as an action that on rare occasions may fail). The agent maintains as its set of contingent beliefs the most plausible, or simplest, picture of the world, consistent with its beliefs and actions it believes it executed; yet it may modify these in light of later information.

Hector J. Levesque

Papers

The substitutional framework for sorted deduction: fundamental results on hybrid reasoning

Rules in Production Systems

A Logical Theory of Robot Localization.

Adding DL-Lite TBoxes to Proper Knowledge Bases

An Epistemic Approach to Nondeterminism: Believing in the Simplest Course of Events