We introduce a new language representation model called BERT, which stands for Bidirectional Encoder Representations from Transformers. Unlike recent language representation models, BERT is designed to pre-train deep bidirectional representations from unlabeled text by jointly conditioning on both left and right context in all layers. As a result, the pre-trained BERT model can be fine-tuned with just one additional output layer to create state-of-the-art models for a wide range of tasks, such as question answering and language inference, without substantial task-specific architecture modifications. 
BERT is conceptually simple and empirically powerful. It obtains new state-of-the-art results on eleven natural language processing tasks, including pushing the GLUE score to 80.5% (7.7% point absolute improvement), MultiNLI accuracy to 86.7% (4.6% absolute improvement), SQuAD v1.1 question answering Test F1 to 93.2 (1.5 point absolute improvement) and SQuAD v2.0 Test F1 to 83.1 (5.1 point absolute improvement).

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

We introduce a new language representation model called BERT, which stands for Bidirectional Encoder Representations from Transformers. Unlike recent language representation models (Peters et al., 2018a; Radford et al., 2018), BERT is designed to pre-train deep bidirectional representations from unlabeled text by jointly conditioning on both left and right context in all layers. As a result, the pre-trained BERT model can be fine-tuned with just one additional output layer to create state-of-the-art models for a wide range of tasks, such as question answering and language inference, without substantial task-specific architecture modifications. BERT is conceptually simple and empirically powerful. It obtains new state-of-the-art results on eleven natural language processing tasks, including pushing the GLUE score to 80.5 (7.7 point absolute improvement), MultiNLI accuracy to 86.7% (4.6% absolute improvement), SQuAD v1.1 question answering Test F1 to 93.2 (1.5 point absolute improvement) and SQuAD v2.0 Test F1 to 83.1 (5.1 point absolute improvement).

Language model pretraining has led to significant performance gains but careful comparison between different approaches is challenging. Training is computationally expensive, often done on private datasets of different sizes, and, as we will show, hyperparameter choices have significant impact on the final results. We present a replication study of BERT pretraining (Devlin et al., 2019) that carefully measures the impact of many key hyperparameters and training data size. We find that BERT was significantly undertrained, and can match or exceed the performance of every model published after it. Our best model achieves state-of-the-art results on GLUE, RACE and SQuAD. These results highlight the importance of previously overlooked design choices, and raise questions about the source of recently reported improvements. We release our models and code.

/pdf/roberta-a-robustly-optimized-bert-pretraining-approach-2rj0bdyhim.pdf

RoBERTa: A Robustly Optimized BERT Pretraining Approach

A translation approach to portable ontology specifications

Recent work has demonstrated substantial gains on many NLP tasks and benchmarks by pre-training on a large corpus of text followed by fine-tuning on a specific task. While typically task-agnostic in architecture, this method still requires task-specific fine-tuning datasets of thousands or tens of thousands of examples. By contrast, humans can generally perform a new language task from only a few examples or from simple instructions - something which current NLP systems still largely struggle to do. Here we show that scaling up language models greatly improves task-agnostic, few-shot performance, sometimes even reaching competitiveness with prior state-of-the-art fine-tuning approaches. Specifically, we train GPT-3, an autoregressive language model with 175 billion parameters, 10x more than any previous non-sparse language model, and test its performance in the few-shot setting. For all tasks, GPT-3 is applied without any gradient updates or fine-tuning, with tasks and few-shot demonstrations specified purely via text interaction with the model. GPT-3 achieves strong performance on many NLP datasets, including translation, question-answering, and cloze tasks, as well as several tasks that require on-the-fly reasoning or domain adaptation, such as unscrambling words, using a novel word in a sentence, or performing 3-digit arithmetic. At the same time, we also identify some datasets where GPT-3's few-shot learning still struggles, as well as some datasets where GPT-3 faces methodological issues related to training on large web corpora. Finally, we find that GPT-3 can generate samples of news articles which human evaluators have difficulty distinguishing from articles written by humans. We discuss broader societal impacts of this finding and of GPT-3 in general.

/pdf/language-models-are-few-shot-learners-2fg8gvia7m.pdf

Language Models are Few-Shot Learners

Only-knowing was originally introduced by Levesque to capture the beliefs of an agent in the sense that its knowledge base is all the agent knows. When a knowledge base contains defaults Levesque also showed an exact correspondence between only-knowing and autoepistemic logic. Later these results were extended by Lakemeyer and Levesque to also capture a variant of autoepistemic logic proposed by Konolige and Reiter's default logic. One of the benefits of such an approach is that various nonmonotonic formalisms can be compared within a single monotonic logic leading, among other things, to the first axiom system for default logic. In this paper, we will bring another large class of nonmonotonic systems, which were first studied by McDermott and Doyle, into the only-knowing fold. Among other things, we will provide the first possible-world semantics for such systems, providing a new perspective on the nature of modal approaches to nonmonotonic reasoning.

Only-knowing meets nonmonotonic modal logic

This work develops an approach to efficient reasoning in first-order knowledge bases with incomplete information. We build on Levesque's proper knowledge bases approach, which supports limited incomplete knowledge in the form of a possibly infinite set of positive or negative ground facts. We propose a generalization which allows these facts to involve unknown individuals, as in the work on labeled null values in databases. Dealing with such unknown individuals has been shown to be a key feature in the database literature on data integration and data exchange. In this way, we obtain one of the most expressive first-order open-world settings for which reasoning can still be done efficiently by evaluation, as in relational databases. We show the soundness of the reasoning procedure and its completeness for queries in a certain normal form.

Efficient reasoning in proper knowledge bases with unknown individuals

Golog is a new programming language based on a theory of action in the situation calculus that can be used to develop multi-agent applications. The Golog interpreter automatically maintains an explicit model of the agent's environment on the basis of user supplied axioms about the preconditions and effects of actions and the initial state of the environment. This allows agent programs to query the state of the environment and consider the effects of various possible courses of action before deciding how to act. This paper discusses a substantial multi-agent application developed in Golog: a system to support personal banking over computer networks. We describe the overall system and provide more details on the agent that assists the user in responding to changes in his financial situation. The advantages and limitations of Golog for developing multi-agent applications are discussed and various extensions are suggested.

An Experiment in Using Golog to Build a Personal Banking Assistant

Nondeterminism is pervasive in all but the simplest action domains: an agent may flip a coin or pick up a different object than intended, or an action may fail and may fail in different ways. In this paper we provide a qualitative theory of nondeterminism. The account is based on an epistemic extension to the situation calculus that accommodates sensing actions. Our position is that nondeterminism is an epistemic phenomenon, and that the world is most usefully regarded as deterministic. Nondeterminism arises from an agent's limited awareness and perception. The account offers several advantages: an agent has a set of categorical (as opposed to probabilistic) beliefs, yet can deal with equally-likely outcomes (such as in flipping a fair coin) or with outcomes of differing plausibility (such as an action that may on rare occasion fail).

/pdf/a-formal-account-of-nondeterministic-and-failed-actions-stij3y6saq.pdf

A formal account of nondeterministic and failed actions

Apar t from the conceptual" gains o f f e r ed by having a taxonomy of c l a s s e s , the main purpose of the ISA h i e r a r c h y is the i n h e r i t a n c e o f d e f i n i t i o n a l p r o p e r t i e s which is a g e n e r a l i z a t i o n of the d e f a u l t mechanism t h a t c a p i t a l i z e s on the expected s i m i l a r i t y between a subclass and a superc l a s s . The PARTOF h i e r a r c h y , on the o ther hand, i nvo l ves the compos i t ion of groups o f ob jec ts i n t o f u n c t i o n a l u n i t s . Such u n i t s are c a l l e d s t r u c t u r e s and the ob jec ts t ha t c o n s t i t u t e them, t h e i r p a r t s . For c l asses , the p a r t s are v a r i a b l e s ( s l o t s ) which are bound to ob jec t s ( s l o t f i l l e r s ) whenever the c lass i s i n s t a n t i a t e d The s l o t s may have d e f a u l t va lues and r e l a t i o n s c a l l e d dependencies a t t a c h ing them to o ther ob jec ts and hav ing many poss ib l e p rocedura l i n t e r p r e t a t i o n s such as de te rmin ing the a c c e p t a b i l i t y of a s l o t f i l l e r o r i n f e r r i n g i t s va lue . Whereas r e l a t i o n s represent the a s s e r t i o n a l p roper t i e s o f a c l a s s , s l o t s represent the d e f i n i t i o n a l p r o p e r t i e s . Consequent ly , the PARTOF h ie ra r chy determines the i nhe r i t a b l e p r o p e r t i e s of a c lass w h i l e the ISA h ie ra r chy s p e c i f i e s the i n h e r i t a n c e pa ths . The metaclass "program" ( the c lass of a l l programs) can be thought of as a s t r u c t u r e having p r e r e q u i s i t e , e f f e c t e t c . as s l o t s and each program f i l l s these s l o t s by p r o v i d i n g p a r t i c u l a r p r e r e q u i s i t e s e t c . This a l lows programs to be i n t e g r a t e d i n t o the two h i e r a r c h i e s and b e n e f i t f rom the o r g a n i z a t i o n ( e . g . i n h e r i t a n c e ) l i k e any o ther c l a s s . S i m i l a r l y , the metaclass " c l a s s " is a s t r u c t u r e whose s l o t s rep re sent the fou r bas ic programs f o r c lasses and the semantics of each c lass are de f i ned by f i l l i n g these s l o t s by s p e c i f i c programs. The d e f a u l t va lues f o r these s l o t s (and t h e : f o u r s i m i l a r ones i n " r e l a t i o n " ) c o n s t i t u t e the s tandard behav iour o f c lasses and r e l a t i o n s p r o v i d i n g the p r i m i t i v e s in terms o f which e v e r y t h i n g e lse can be de f i ned and l i n k i n g the fo rma l i sm to a p h y s i c a l imp lementa t ion . References [Levesque 77] Levesque, H. "A p rocedu ra l approach to semantic n e t w o r k s " , TR-105, Dept. of Computer Sc ience, U. of To ron to , 1977. [Hayes 74] Hayes, P .J . "Some pro blems and non-problems in r e p r e s e n t a t i o n t h e o r y " , Proc. AISB summer conference,1974. [ A b r i a l 74] A b r i a l , J .R. "Data semant ics" in Data Management Systems, K l imb ie and Koffeman ( e d s . ) , N o r t h H o l l a n d , 1974.

/pdf/an-overview-of-a-procedural-approach-to-semantic-networks-55fxmyg27l.pdf

Hector J. Levesque

Papers

Only-knowing meets nonmonotonic modal logic

Efficient reasoning in proper knowledge bases with unknown individuals

An Experiment in Using Golog to Build a Personal Banking Assistant

A formal account of nondeterministic and failed actions

An overview of a procedural approach to semantic networks