Recent work has demonstrated substantial gains on many NLP tasks and benchmarks by pre-training on a large corpus of text followed by fine-tuning on a specific task. While typically task-agnostic in architecture, this method still requires task-specific fine-tuning datasets of thousands or tens of thousands of examples. By contrast, humans can generally perform a new language task from only a few examples or from simple instructions - something which current NLP systems still largely struggle to do. Here we show that scaling up language models greatly improves task-agnostic, few-shot performance, sometimes even reaching competitiveness with prior state-of-the-art fine-tuning approaches. Specifically, we train GPT-3, an autoregressive language model with 175 billion parameters, 10x more than any previous non-sparse language model, and test its performance in the few-shot setting. For all tasks, GPT-3 is applied without any gradient updates or fine-tuning, with tasks and few-shot demonstrations specified purely via text interaction with the model. GPT-3 achieves strong performance on many NLP datasets, including translation, question-answering, and cloze tasks, as well as several tasks that require on-the-fly reasoning or domain adaptation, such as unscrambling words, using a novel word in a sentence, or performing 3-digit arithmetic. At the same time, we also identify some datasets where GPT-3's few-shot learning still struggles, as well as some datasets where GPT-3 faces methodological issues related to training on large web corpora. Finally, we find that GPT-3 can generate samples of news articles which human evaluators have difficulty distinguishing from articles written by humans. We discuss broader societal impacts of this finding and of GPT-3 in general.

/pdf/language-models-are-few-shot-learners-2fg8gvia7m.pdf

Language Models are Few-Shot Learners

Transfer learning, where a model is first pre-trained on a data-rich task before being fine-tuned on a downstream task, has emerged as a powerful technique in natural language processing (NLP). The effectiveness of transfer learning has given rise to a diversity of approaches, methodology, and practice. In this paper, we explore the landscape of transfer learning techniques for NLP by introducing a unified framework that converts all text-based language problems into a text-to-text format. Our systematic study compares pre-training objectives, architectures, unlabeled data sets, transfer approaches, and other factors on dozens of language understanding tasks. By combining the insights from our exploration with scale and our new ``Colossal Clean Crawled Corpus'', we achieve state-of-the-art results on many benchmarks covering summarization, question answering, text classification, and more. To facilitate future work on transfer learning for NLP, we release our data set, pre-trained models, and code.

Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer

Part I. Experimental Studies: 2. Experiment in psychology 3. Experiments on perceiving III Experiments on imaging 4-8. Experiments on remembering: (a) The method of description (b) The method of repeated reproduction (c) The method of picture writing (d) The method of serial reproduction (e) The method of serial reproduction picture material 9. Perceiving, recognizing, remembering 10. A theory of remembering 11. Images and their functions 12. Meaning Part II. Remembering as a Study in Social Psychology: 13. Social psychology 14. Social psychology and the matter of recall 15. Social psychology and the manner of recall 16. Conventionalism 17. The notion of a collective unconscious 18. The basis of social recall 19. A summary and some conclusions.

Remembering. A Study in Experimental and Social Psychology, Cambridge (University Press) 1964.

Recent progress in natural language processing has been driven by advances in both model architecture and model pretraining. Transformer architectures have facilitated building higher-capacity models and pretraining has made it possible to effectively utilize this capacity for a wide variety of tasks. \textit{Transformers} is an open-source library with the goal of opening up these advances to the wider machine learning community. The library consists of carefully engineered state-of-the art Transformer architectures under a unified API. Backing this library is a curated collection of pretrained models made by and available for the community. \textit{Transformers} is designed to be extensible by researchers, simple for practitioners, and fast and robust in industrial deployments. The library is available at \url{this https URL}.

HuggingFace's Transformers: State-of-the-art Natural Language Processing.

A wide variety of media can be used in learning, including distance learning, such as print, lectures, conference sections, tutors, pictures, video, sound, and computers. Any one instance of distance learning will make choices among these media, perhaps using several.

Multimedia in Learning

The Army Excellence in Leadership (AXL) project at the University of Southern California’s Institute for Creative Technologies is aimed at supporting the acquisition of tacit knowledge of military leadership through the development of compelling filmed narratives of leadership scenarios and interactive training technologies. The approach taken in the AXL project is to leverage the best practices of case-method teaching and use Hollywood storytelling techniques to create fictional case studies (as filmed media) addressing specific leadership issues. In addition to authoring compelling cases for analysis, we have developed software prototypes that instantiate the case-method teaching approach. These systems engage individual trainees in human-computer dialogues that are focused on the leadership issues that have been embedded in the fictional cases.

https://ict.usc.edu/pubs/LEARNING%20THE%20LESSONS%20OF%20LEADERSHIP%20EXPERIENCE-%20TOOLS%20FOR%20INTERACTIVE%20CASE%20METHOD%20ANALYSIS.pdf

Learning the lessons of leadership experience: tools for interactive case method analysis

Among the most interesting ways that people share knowledge is through the telling of stories, i.e. first-person narratives about real-life experiences. Millions of these stories appear in Internet weblogs, offering a potentially valuable resource for future knowledge management and training applications. In this paper we describe efforts to automatically capture stories from Internet weblogs by extracting them using statistical text classification techniques. We evaluate the precision and recall performance of competing approaches. We describe the large-scale application of story extraction technology to Internet weblogs, producing a corpus of stories with over a billion words.

/pdf/automated-story-capture-from-internet-weblogs-2y6vy4pzxf.pdf

Automated story capture from internet weblogs

The research described here is part of a larger effort, first, to construct formal theories of a broad range of aspects of commonsense psychology, including knowledge management, the envisionment of possible courses of events, and goal-directed behavior, and, second, to link them to the English lexicon. We have identified the most common words and phrases for describing emotions in English. In this paper we describe a formalization of people’s implicit theory of how emotions mediate between what they experience and what they do. We then sketch out effort to write rules that link the theory with words and phrases in the emotional lexicon.

/pdf/the-deep-lexical-semantics-of-emotions-3iy1elr1ru.pdf

The Deep Lexical Semantics of Emotions

A method and apparatus (Fig. 1) is disclosed for advanced leadership training simulation wherein the simulation teaches skills in leadership and related topics through an Internet-based distance-learning architecture. The distance-learning features link trainees at remote locations into a single collaborative experience via computer networks. Instructional storylines are created and programmed into a computer and then delivered as a simulated but realistic story to one or more participants. The participants' reactions are monitored and compared with expected results. The storyline may be altered in response to the participants' responses, and synthetic characters may be generated to act as automated participants or coaches. Constructive feedback is provided to the participants during or after the simulation.

Method and apparatus for advanced leadership training simulation

This paper describes the value of capitalizing on Hollywood storytelling techniques in the design of story-based training applications built around branching storylines. After reviewing the design of Outcome-Driven Simulations and the technical aspects of our application prototype, we describe storytelling techniques that greatly improve the level of user engagement in training simulations based on this design. These techniques concern the overall development of the story, the use of a story arc, the critical decisions in a story, notions of pay off and climax, dramatic sequences, character bibles, characters as a Greek chorus, and the significance of consequences and outcomes. Examples of each of these storytelling techniques are given in the context of the ICT Leaders Project, a prototype leadership development application for the US

/pdf/bringing-hollywood-storytelling-techniques-to-branching-3dt6txqtfm.pdf

Andrew S. Gordon

Papers

Learning the lessons of leadership experience: tools for interactive case method analysis

Automated story capture from internet weblogs

The Deep Lexical Semantics of Emotions

Method and apparatus for advanced leadership training simulation

Bringing Hollywood Storytelling Techniques to Branching Storylines for Training Applications