We introduce a new language representation model called BERT, which stands for Bidirectional Encoder Representations from Transformers. Unlike recent language representation models, BERT is designed to pre-train deep bidirectional representations from unlabeled text by jointly conditioning on both left and right context in all layers. As a result, the pre-trained BERT model can be fine-tuned with just one additional output layer to create state-of-the-art models for a wide range of tasks, such as question answering and language inference, without substantial task-specific architecture modifications. 
BERT is conceptually simple and empirically powerful. It obtains new state-of-the-art results on eleven natural language processing tasks, including pushing the GLUE score to 80.5% (7.7% point absolute improvement), MultiNLI accuracy to 86.7% (4.6% absolute improvement), SQuAD v1.1 question answering Test F1 to 93.2 (1.5 point absolute improvement) and SQuAD v2.0 Test F1 to 83.1 (5.1 point absolute improvement).

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

We introduce a new language representation model called BERT, which stands for Bidirectional Encoder Representations from Transformers. Unlike recent language representation models (Peters et al., 2018a; Radford et al., 2018), BERT is designed to pre-train deep bidirectional representations from unlabeled text by jointly conditioning on both left and right context in all layers. As a result, the pre-trained BERT model can be fine-tuned with just one additional output layer to create state-of-the-art models for a wide range of tasks, such as question answering and language inference, without substantial task-specific architecture modifications. BERT is conceptually simple and empirically powerful. It obtains new state-of-the-art results on eleven natural language processing tasks, including pushing the GLUE score to 80.5 (7.7 point absolute improvement), MultiNLI accuracy to 86.7% (4.6% absolute improvement), SQuAD v1.1 question answering Test F1 to 93.2 (1.5 point absolute improvement) and SQuAD v2.0 Test F1 to 83.1 (5.1 point absolute improvement).

Language model pretraining has led to significant performance gains but careful comparison between different approaches is challenging. Training is computationally expensive, often done on private datasets of different sizes, and, as we will show, hyperparameter choices have significant impact on the final results. We present a replication study of BERT pretraining (Devlin et al., 2019) that carefully measures the impact of many key hyperparameters and training data size. We find that BERT was significantly undertrained, and can match or exceed the performance of every model published after it. Our best model achieves state-of-the-art results on GLUE, RACE and SQuAD. These results highlight the importance of previously overlooked design choices, and raise questions about the source of recently reported improvements. We release our models and code.

/pdf/roberta-a-robustly-optimized-bert-pretraining-approach-2rj0bdyhim.pdf

RoBERTa: A Robustly Optimized BERT Pretraining Approach

A translation approach to portable ontology specifications

Recent work has demonstrated substantial gains on many NLP tasks and benchmarks by pre-training on a large corpus of text followed by fine-tuning on a specific task. While typically task-agnostic in architecture, this method still requires task-specific fine-tuning datasets of thousands or tens of thousands of examples. By contrast, humans can generally perform a new language task from only a few examples or from simple instructions - something which current NLP systems still largely struggle to do. Here we show that scaling up language models greatly improves task-agnostic, few-shot performance, sometimes even reaching competitiveness with prior state-of-the-art fine-tuning approaches. Specifically, we train GPT-3, an autoregressive language model with 175 billion parameters, 10x more than any previous non-sparse language model, and test its performance in the few-shot setting. For all tasks, GPT-3 is applied without any gradient updates or fine-tuning, with tasks and few-shot demonstrations specified purely via text interaction with the model. GPT-3 achieves strong performance on many NLP datasets, including translation, question-answering, and cloze tasks, as well as several tasks that require on-the-fly reasoning or domain adaptation, such as unscrambling words, using a novel word in a sentence, or performing 3-digit arithmetic. At the same time, we also identify some datasets where GPT-3's few-shot learning still struggles, as well as some datasets where GPT-3 faces methodological issues related to training on large web corpora. Finally, we find that GPT-3 can generate samples of news articles which human evaluators have difficulty distinguishing from articles written by humans. We discuss broader societal impacts of this finding and of GPT-3 in general.

/pdf/language-models-are-few-shot-learners-2fg8gvia7m.pdf

Language Models are Few-Shot Learners

Publisher Summary
This chapter talks about default reasoning in detail and in logical terms and also examines four different logical formalisms forit. . Although each of them does the job in many cases, each formalism has drawbacks of one sort or another. Getting a logical account of default reasoning that is simple and applicable in broad term, and intuitively correct remains an open problem. In fact, because so much of what is to be known involves default reasoning, it is perhaps the open problem in the whole area of knowledge representation. Default Reasoning is based on the assumption that is in general terms known as the closed-world assumption (CWA). Ordinary deductive reasoning is monotonic, that is to say that new facts can only produce additional beliefs. In contrast, default reasoning is nonmonotonic. The simple formalization of default reasoning is considered in the chapter.

Chapter 11 – Defaults

Publisher Summary
This chapter outlines the basic principles of knowledge representation and decides an initial representation language. The task of the chapter is to create a knowledge base (KB) that contains appropriate entailments in first-order logic (FOL). In creating a KB it is a good idea to start with the set of domain-dependent predicates and functions that provide the basis for the statement of facts about the KB's domain. The most obvious place to start is with the named individuals who are the actors in human drama. In FOL, these would be represented by constant symbols, like mary Jones, johnQSmith, and so on. The FOL language gives the basic tools for representing facts in a domain, but in many cases there is a great deal of flexibility that can be exercised in mapping objects in that domain onto predicates and functions. Another class of named individuals would be the legal entities that have their own identities, such as corporations (faultylnsuranceCompany), governments (evilvilleTownCouncil), and restaurants (theRackAndRollRestaurant.

Chapter 3 – Expressing Knowledge

Publisher Summary
This chapter discusses the knowledge representation and reasoning hypothesis. According to the knowledge representation hypothesis, intelligent behavior ultimately depends on explicitly represented knowledge. No other design strategy as yet seems plausible to explain behavior, or have it depend in a flexible way on what one is told, or to isolate the assumptions that govern it, or to achieve any number of other desirable traits. However, KR by itself does not solve anything unless a system is able to reason effectively with what it has explicitly represented. Specifically, if reasoning needs to be performed automatically as a part of a larger task, it must be dependable both in terms of what it calculates and how long it takes. Thus, KR is the study of what information can be extracted, in a computationally dependable way, from what forms of represented knowledge. In other words, it investigates the area within the confines of the KR hypothesis.

Knowledge representation and reasoning1

Ray dedicated his life to his research with the wonder of a child, the fearlessness of an explorer, the precision of a mathematician, and the tirelessness of a researcher who found shallowness and confusion intolerable. He leaves a legacy of groundbreaking, deep insights that have changed the course of AI.

/pdf/in-memory-of-ray-reiter-1939-2002-1jj9kb6bgh.pdf

In Memory of Ray Reiter (1939-2002)

Publisher Summary
This chapter deals with the representation techniques that look more directly towards the aspects of objects and categories than frames did. During focusing on the more declarative aspects of an object-oriented representation the chapter analyzes the concepts such as predicates and entailment from First-order logic (FOL). Traditional first-order logic does not provide any tools for dealing with compound predicates of this sort. In a sense, the only noun phrases in FOL are the nouns. But given the prominence and naturalness of such constructs in natural language, it is worthwhile to consider knowledge representation machinery that does provide such tools. Because a logic that allows to manipulate complex predicates is expected be working mainly with descriptions, that is known as system based on the ideas of description logic (DL). There are four types of legal syntactic expressions in DL including constants, roles (both seen earlier), concepts, and sentences. These expressions are further described in much detail in the chapter.

Hector J. Levesque

Papers

Chapter 11 – Defaults

Chapter 3 – Expressing Knowledge

Knowledge representation and reasoning1

In Memory of Ray Reiter (1939-2002)

Chapter 9 – Structured Descriptions