We introduce a new language representation model called BERT, which stands for Bidirectional Encoder Representations from Transformers. Unlike recent language representation models, BERT is designed to pre-train deep bidirectional representations from unlabeled text by jointly conditioning on both left and right context in all layers. As a result, the pre-trained BERT model can be fine-tuned with just one additional output layer to create state-of-the-art models for a wide range of tasks, such as question answering and language inference, without substantial task-specific architecture modifications. 
BERT is conceptually simple and empirically powerful. It obtains new state-of-the-art results on eleven natural language processing tasks, including pushing the GLUE score to 80.5% (7.7% point absolute improvement), MultiNLI accuracy to 86.7% (4.6% absolute improvement), SQuAD v1.1 question answering Test F1 to 93.2 (1.5 point absolute improvement) and SQuAD v2.0 Test F1 to 83.1 (5.1 point absolute improvement).

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

We introduce a new language representation model called BERT, which stands for Bidirectional Encoder Representations from Transformers. Unlike recent language representation models (Peters et al., 2018a; Radford et al., 2018), BERT is designed to pre-train deep bidirectional representations from unlabeled text by jointly conditioning on both left and right context in all layers. As a result, the pre-trained BERT model can be fine-tuned with just one additional output layer to create state-of-the-art models for a wide range of tasks, such as question answering and language inference, without substantial task-specific architecture modifications. BERT is conceptually simple and empirically powerful. It obtains new state-of-the-art results on eleven natural language processing tasks, including pushing the GLUE score to 80.5 (7.7 point absolute improvement), MultiNLI accuracy to 86.7% (4.6% absolute improvement), SQuAD v1.1 question answering Test F1 to 93.2 (1.5 point absolute improvement) and SQuAD v2.0 Test F1 to 83.1 (5.1 point absolute improvement).

Language model pretraining has led to significant performance gains but careful comparison between different approaches is challenging. Training is computationally expensive, often done on private datasets of different sizes, and, as we will show, hyperparameter choices have significant impact on the final results. We present a replication study of BERT pretraining (Devlin et al., 2019) that carefully measures the impact of many key hyperparameters and training data size. We find that BERT was significantly undertrained, and can match or exceed the performance of every model published after it. Our best model achieves state-of-the-art results on GLUE, RACE and SQuAD. These results highlight the importance of previously overlooked design choices, and raise questions about the source of recently reported improvements. We release our models and code.

/pdf/roberta-a-robustly-optimized-bert-pretraining-approach-2rj0bdyhim.pdf

RoBERTa: A Robustly Optimized BERT Pretraining Approach

A translation approach to portable ontology specifications

Recent work has demonstrated substantial gains on many NLP tasks and benchmarks by pre-training on a large corpus of text followed by fine-tuning on a specific task. While typically task-agnostic in architecture, this method still requires task-specific fine-tuning datasets of thousands or tens of thousands of examples. By contrast, humans can generally perform a new language task from only a few examples or from simple instructions - something which current NLP systems still largely struggle to do. Here we show that scaling up language models greatly improves task-agnostic, few-shot performance, sometimes even reaching competitiveness with prior state-of-the-art fine-tuning approaches. Specifically, we train GPT-3, an autoregressive language model with 175 billion parameters, 10x more than any previous non-sparse language model, and test its performance in the few-shot setting. For all tasks, GPT-3 is applied without any gradient updates or fine-tuning, with tasks and few-shot demonstrations specified purely via text interaction with the model. GPT-3 achieves strong performance on many NLP datasets, including translation, question-answering, and cloze tasks, as well as several tasks that require on-the-fly reasoning or domain adaptation, such as unscrambling words, using a novel word in a sentence, or performing 3-digit arithmetic. At the same time, we also identify some datasets where GPT-3's few-shot learning still struggles, as well as some datasets where GPT-3 faces methodological issues related to training on large web corpora. Finally, we find that GPT-3 can generate samples of news articles which human evaluators have difficulty distinguishing from articles written by humans. We discuss broader societal impacts of this finding and of GPT-3 in general.

/pdf/language-models-are-few-shot-learners-2fg8gvia7m.pdf

Language Models are Few-Shot Learners

What can artificial intelligence teach us about the mind? If AI's underlying concept is that thinking is a computational process, then how can computation illuminate thinking? It's a timely question. AI is all the rage, and the buzziest AI buzz surrounds adaptive machine learning: computer systems that learn intelligent behavior from massive amounts of data. This is what powers a driverless car, for example. In this book, Hector Levesque shifts the conversation to "good old fashioned artificial intelligence," which is based not on heaps of data but on understanding commonsense intelligence. This kind of artificial intelligence is equipped to handle situations that depart from previous patterns -- as we do in real life, when, for example, we encounter a washed-out bridge or when the barista informs us there's no more soy milk. Levesque considers the role of language in learning. He argues that a computer program that passes the famous Turing Test could be a mindless zombie, and he proposes another way to test for intelligence -- the Winograd Schema Test, developed by Levesque and his colleagues. "If our goal is to understand intelligent behavior, we had better understand the difference between making it and faking it," he observes. He identifies a possible mechanism behind common sense and the capacity to call on background knowledge: the ability to represent objects of thought symbolically. As AI migrates more and more into everyday life, we should worry if systems without common sense are making decisions where common sense is needed.

Common Sense, the Turing Test, and the Quest for Real AI

Touretzky [1984] proposed a formalism for nonmonotonic multiple inheritance reasoning that is sound in the presence of ambiguities and redundant links. We show that Touretzky's inheritance notion is NP-hard, and thus, provided P ≠ NP, computationally intractable. This result holds even when one only considers unambiguous, totally acyclic inheritance networks. A direct consequence of this result is that the conditioning strategy proposed by Touretzky to allow for fast parallel inference is also intractable. Therefore, it follows that nonmonotonic multiple inheritance hierarchies, although compact representations, may not allow for efficient retrieval of information as has been suggested in attempts to use such hierarchies, e.g., in NETL [Fahlman, 1979]. We also analyze the influence of various design choices made by Touretzky. We show that all versions of downward (coupled) inheritance, i.e., on-path or off-path preemption and skeptical or credulous reasoning, are intractable. However, tractability can be achieved when using upward (decoupled) inheritance. Thus, the main source of intractability in path-based inheritance formalisms is the downward (coupled) reasoning.

/pdf/the-tractability-of-path-based-inheritance-e6q0ky3j0n.pdf

The tractability of path-based inheritance

The range of domains and tasks for “knowledgebased systems” has been expanding at a furious pace. As we move away from trivial domains, such as the “blocks world”, the demands on knowledge representation systems used by expert programs are becoming more extreme. For one thing, the domains themselves are getting so complex that specialized technical vocabularies are unavoidable; consequently, the issue of a system talking with un expert in ki8 own ianguage cannot be ignored. For another, tasks such as medical diagnosis, scene analysis, speech understanding, and game playing all have as a central feature an incrementally evolving model representing probably incomplete knowledge of part of the task domain. In this paper, we explore some of the impact of these two critical issues-complexity and incompletenesson knowledge representation systems. We review some aspects of current representation research that offer a foundation for coping with these problems, and finally suggest a way of integrating these ideas into a powerful, practical knowledge representation paradigm.

/pdf/competence-in-knowledge-representation-2furcwi50r.pdf

Competence in knowledge representation

Most agents can acquire information about their environments as they operate A good plan for such an agent is one that not only achieves the goal, but is also executable, ie, ensures that the agent has enough information at every step to know what to do next In this paper, we present a formal account of what it means for an agent to know how to execute a plan and to be able to achieve a goal Such a theory is a prerequisite for producing specifications of planners for agents that can acquire information at run time It is also essential to account for cooperation among agents Our account is more general than previous proposals, correctly handles programs containing loops, and incorporates a solution to the frame problem It can also be used to prove programs containing sensing actions correct

/pdf/ability-and-knowing-how-in-the-situation-calculus-16iddmlfps.pdf

Ability and knowing how in the situation calculus

The formal representation issues underlying the use of incomplete knowledge bases are investigated. It is argued that to effectively deal with incomplete knowledge, a system must first be able to determine where the incompleteness lies. This has two consequences: first, the knowledge base must have knowledge about its own incompleteness and second, it must be possible to ask or tell the knowledge base about its incompleteness. A formal model of knowledge bases with these abilities is presented. The characterization is in terms of a logical language that can refer both to application domains and to what knowledge bases might know about such domains. This language is also used to formulate questions, statements and defaults for a knowledge base. A semantic and proof-theoretic analysis is provided not only for the language itself, but also for the operations of answering questions, acquiring knowledge and assuming defaults.

Hector J. Levesque

Papers

Common Sense, the Turing Test, and the Quest for Real AI

The tractability of path-based inheritance

Competence in knowledge representation

Ability and knowing how in the situation calculus

A formal treatment of incomplete knowledge bases