Recent work has demonstrated substantial gains on many NLP tasks and benchmarks by pre-training on a large corpus of text followed by fine-tuning on a specific task. While typically task-agnostic in architecture, this method still requires task-specific fine-tuning datasets of thousands or tens of thousands of examples. By contrast, humans can generally perform a new language task from only a few examples or from simple instructions - something which current NLP systems still largely struggle to do. Here we show that scaling up language models greatly improves task-agnostic, few-shot performance, sometimes even reaching competitiveness with prior state-of-the-art fine-tuning approaches. Specifically, we train GPT-3, an autoregressive language model with 175 billion parameters, 10x more than any previous non-sparse language model, and test its performance in the few-shot setting. For all tasks, GPT-3 is applied without any gradient updates or fine-tuning, with tasks and few-shot demonstrations specified purely via text interaction with the model. GPT-3 achieves strong performance on many NLP datasets, including translation, question-answering, and cloze tasks, as well as several tasks that require on-the-fly reasoning or domain adaptation, such as unscrambling words, using a novel word in a sentence, or performing 3-digit arithmetic. At the same time, we also identify some datasets where GPT-3's few-shot learning still struggles, as well as some datasets where GPT-3 faces methodological issues related to training on large web corpora. Finally, we find that GPT-3 can generate samples of news articles which human evaluators have difficulty distinguishing from articles written by humans. We discuss broader societal impacts of this finding and of GPT-3 in general.

/pdf/language-models-are-few-shot-learners-2fg8gvia7m.pdf

Language Models are Few-Shot Learners

Transfer learning, where a model is first pre-trained on a data-rich task before being fine-tuned on a downstream task, has emerged as a powerful technique in natural language processing (NLP). The effectiveness of transfer learning has given rise to a diversity of approaches, methodology, and practice. In this paper, we explore the landscape of transfer learning techniques for NLP by introducing a unified framework that converts all text-based language problems into a text-to-text format. Our systematic study compares pre-training objectives, architectures, unlabeled data sets, transfer approaches, and other factors on dozens of language understanding tasks. By combining the insights from our exploration with scale and our new ``Colossal Clean Crawled Corpus'', we achieve state-of-the-art results on many benchmarks covering summarization, question answering, text classification, and more. To facilitate future work on transfer learning for NLP, we release our data set, pre-trained models, and code.

Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer

Part I. Experimental Studies: 2. Experiment in psychology 3. Experiments on perceiving III Experiments on imaging 4-8. Experiments on remembering: (a) The method of description (b) The method of repeated reproduction (c) The method of picture writing (d) The method of serial reproduction (e) The method of serial reproduction picture material 9. Perceiving, recognizing, remembering 10. A theory of remembering 11. Images and their functions 12. Meaning Part II. Remembering as a Study in Social Psychology: 13. Social psychology 14. Social psychology and the matter of recall 15. Social psychology and the manner of recall 16. Conventionalism 17. The notion of a collective unconscious 18. The basis of social recall 19. A summary and some conclusions.

Remembering. A Study in Experimental and Social Psychology, Cambridge (University Press) 1964.

Recent progress in natural language processing has been driven by advances in both model architecture and model pretraining. Transformer architectures have facilitated building higher-capacity models and pretraining has made it possible to effectively utilize this capacity for a wide variety of tasks. \textit{Transformers} is an open-source library with the goal of opening up these advances to the wider machine learning community. The library consists of carefully engineered state-of-the art Transformer architectures under a unified API. Backing this library is a curated collection of pretrained models made by and available for the community. \textit{Transformers} is designed to be extensible by researchers, simple for practitioners, and fast and robust in industrial deployments. The library is available at \url{this https URL}.

HuggingFace's Transformers: State-of-the-art Natural Language Processing.

A wide variety of media can be used in learning, including distance learning, such as print, lectures, conference sections, tutors, pictures, video, sound, and computers. Any one instance of distance learning will make choices among these media, perhaps using several.

Multimedia in Learning

The utility of formal theories of commonsense reasoning will depend both on their competency in solving problems and on their conceptual coverage. We argue that the problems of coverage and competency can be decoupled and solved with dierent methods for a given commonsense domain. We describe a methodology for identifying the coverage requirements of theories through the large-scale analysis of planning strategies, with further refinements made by collecting and categorizing instances of natural language expressions pertaining to the domain. We demonstrate the eectiveness of this methodology in identifying the representational coverage requirements of theories of the commonsense psychology of human memory. We then apply traditional methods of formalization to produce a formal first-order theory of commonsense memory with a high degree of competency and coverage. 1 Coverage and Competency While much research in commonsense reasoning has been directed at describing axiomatic content theories in specific areas, of equal concern are the research methods that are used to develop these content theories. Davis (1998) reflects back on the methodological problems that have hindered progress and recommends a research program based on microworlds. He argues that the goal of commonsense reasoning research is the generation of competency theories that can answer commonsense problems that people are able to solve. By emphasizing reasoning competency, Davis makes a strong case for focusing on the function of axiomatic theories rather than their form. However, while the representational form may be indeterminate with respect to its function, representation itself has an even broader role to play across the full spectrum of cognitive behavior, beyond the commonsense reasoning functions of explanation, prediction, planning and design. What is needed of commonsense theories is not only competency, but also enough coverage over the breadth of commonsense concepts to enable use in computational models of memory retrieval, language understanding, perception, similarity, among other cognitive functions. A conservative commonsense reasoning researcher might argue that coverage is an additional constraint on an already difficult task, and is best addressed after suitable competency theories have been put forth. We argue that without addressing the issue of coverage first, competency theories will be intolerant of elaboration and dicult to integrate with each other or within larger cognitive models. This paper presents a new methodology for authoring formal commonsense theories. The basis of our approach is the tenet that the problems of coverage and competency should be decoupled and addressed by entirely dierent methods. Our approach begins by outlining the coverage requirements of commonsense theories through the analysis of a corpus of strategies. These requirements are elaborated to handle distinctions made in natural language, as evidenced through the analysis of large English text corpora. We then address the specification of a formal notation (here, first-order predicate calculus) and of a full axiomatic theory. Section 2 of this paper describes the methods used to solve the coverage problem in the domains of commonsense psychology. Section 3 elaborates on the role of natural language in refining these representations, with an example domain of the commonsense psychology of memory. Section 4 presents a formal, axiomatic theory of the com

/pdf/coverage-and-competency-in-formal-theories-a-commonsense-3hiagkuows.pdf

Coverage and Competency in Formal Theories: A Commonsense Theory of Memory

We investigate new interfaces that allow users to specify topics of interest in streams of weblog stories by providing relevance feedback to a search algorithm. Noting that weblog stories often contain photographs taken by the blogger during the course of the narrated events, we investigate whether these photographs can serve as a proxy for the whole post when users are making judgments as to the post's relevance. We developed a new story annotation interface for collecting relevance feedback with three variations: users are presented either with the full post as it appears in a weblog, an embedded photograph, or only the title of the post. We describe a user evaluation that compares annotation time, quality, and subjective user experience across each of these three conditions. The results show that relevance judgments based on embedded photographs or titles are far less accurate than when reading the whole weblog post, but the time required to acquire an accurate model of the user's topic interest is greatly reduced.

/pdf/collecting-relevance-feedback-on-titles-and-photographs-in-250yv306bl.pdf

Collecting relevance feedback on titles and photographs in weblog posts

https://people.ict.usc.edu/~gordon/publications/AIJ04.PDF

The representation of planning strategies

: The Army Excellence in Leadership (AXL) system is an online interactive system for delivering multimedia case method instruction. The intent of the AXL system is to develop leaders with greater interpersonal competence and cultural awareness. To achieve this goal. the AXL research program combines the case method of instruction with Hollywood techniques and interactive online technology. This report describes the technological capabilities of the AXL system, as well as the case method pedagogy underlying the AXL approach. The report describes how the AXL system capitalizes on the best practices of traditional case method instruction and addresses some of the limitations of case method instruction. Additionally, this report outlines how the instructional content in the AXL system relates to the leadership competencies outlined in Army leadership doctrine (FM 6-22) and the tacit knowledge framework for military leaders developed by Sternberg et al. (2000).

http://www.au.af.mil/au/awc/awcgate/army/tr1226.pdf

Learning the Lessons of Leadership: Case Method Teaching with Interactive Computer-Based Tools and Film-Based Cases

In this paper we describe the architecture of an interactive story generation system where a human and computer each take turns writing sentences of an emerging narrative. Each turn begins with the user adding a sentence to the story, where the computer responds with a sentence of its own that continues what has been written so far. Rather than generating the next sentence from scratch, the computer selects the next sentence from a corpus of tens of millions of narrative sentences extracted from Internet weblogs. We compare five different retrieval methods for selecting the most appropriate sentence, and present the results of a user study to determine which of these models produces stories with the highest coherence and overall value.

/pdf/a-comparison-of-retrieval-models-for-open-domain-story-wa3dm7chtn.pdf

Andrew S. Gordon

Papers

Coverage and Competency in Formal Theories: A Commonsense Theory of Memory

Collecting relevance feedback on titles and photographs in weblog posts

The representation of planning strategies

Learning the Lessons of Leadership: Case Method Teaching with Interactive Computer-Based Tools and Film-Based Cases

A Comparison of Retrieval Models for Open Domain Story Generation