Recent work has demonstrated substantial gains on many NLP tasks and benchmarks by pre-training on a large corpus of text followed by fine-tuning on a specific task. While typically task-agnostic in architecture, this method still requires task-specific fine-tuning datasets of thousands or tens of thousands of examples. By contrast, humans can generally perform a new language task from only a few examples or from simple instructions - something which current NLP systems still largely struggle to do. Here we show that scaling up language models greatly improves task-agnostic, few-shot performance, sometimes even reaching competitiveness with prior state-of-the-art fine-tuning approaches. Specifically, we train GPT-3, an autoregressive language model with 175 billion parameters, 10x more than any previous non-sparse language model, and test its performance in the few-shot setting. For all tasks, GPT-3 is applied without any gradient updates or fine-tuning, with tasks and few-shot demonstrations specified purely via text interaction with the model. GPT-3 achieves strong performance on many NLP datasets, including translation, question-answering, and cloze tasks, as well as several tasks that require on-the-fly reasoning or domain adaptation, such as unscrambling words, using a novel word in a sentence, or performing 3-digit arithmetic. At the same time, we also identify some datasets where GPT-3's few-shot learning still struggles, as well as some datasets where GPT-3 faces methodological issues related to training on large web corpora. Finally, we find that GPT-3 can generate samples of news articles which human evaluators have difficulty distinguishing from articles written by humans. We discuss broader societal impacts of this finding and of GPT-3 in general.

/pdf/language-models-are-few-shot-learners-2fg8gvia7m.pdf

Language Models are Few-Shot Learners

Transfer learning, where a model is first pre-trained on a data-rich task before being fine-tuned on a downstream task, has emerged as a powerful technique in natural language processing (NLP). The effectiveness of transfer learning has given rise to a diversity of approaches, methodology, and practice. In this paper, we explore the landscape of transfer learning techniques for NLP by introducing a unified framework that converts all text-based language problems into a text-to-text format. Our systematic study compares pre-training objectives, architectures, unlabeled data sets, transfer approaches, and other factors on dozens of language understanding tasks. By combining the insights from our exploration with scale and our new ``Colossal Clean Crawled Corpus'', we achieve state-of-the-art results on many benchmarks covering summarization, question answering, text classification, and more. To facilitate future work on transfer learning for NLP, we release our data set, pre-trained models, and code.

Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer

Part I. Experimental Studies: 2. Experiment in psychology 3. Experiments on perceiving III Experiments on imaging 4-8. Experiments on remembering: (a) The method of description (b) The method of repeated reproduction (c) The method of picture writing (d) The method of serial reproduction (e) The method of serial reproduction picture material 9. Perceiving, recognizing, remembering 10. A theory of remembering 11. Images and their functions 12. Meaning Part II. Remembering as a Study in Social Psychology: 13. Social psychology 14. Social psychology and the matter of recall 15. Social psychology and the manner of recall 16. Conventionalism 17. The notion of a collective unconscious 18. The basis of social recall 19. A summary and some conclusions.

Remembering. A Study in Experimental and Social Psychology, Cambridge (University Press) 1964.

Recent progress in natural language processing has been driven by advances in both model architecture and model pretraining. Transformer architectures have facilitated building higher-capacity models and pretraining has made it possible to effectively utilize this capacity for a wide variety of tasks. \textit{Transformers} is an open-source library with the goal of opening up these advances to the wider machine learning community. The library consists of carefully engineered state-of-the art Transformer architectures under a unified API. Backing this library is a curated collection of pretrained models made by and available for the community. \textit{Transformers} is designed to be extensible by researchers, simple for practitioners, and fast and robust in industrial deployments. The library is available at \url{this https URL}.

HuggingFace's Transformers: State-of-the-art Natural Language Processing.

A wide variety of media can be used in learning, including distance learning, such as print, lectures, conference sections, tutors, pictures, video, sound, and computers. Any one instance of distance learning will make choices among these media, perhaps using several.

Multimedia in Learning

Digital interactive storytelling (DIS) is a compelling new medium for expressing and communicating ideas that tries to transform a normally passive experience into an active engagement in the creative process. Despite the enormous potential this medium beholds, the cost and complexity of authoring compelling stories primarily driven by user actions is prohibitively expensive in many DIS systems. While the graphical capabilities and physical interaction with these systems have advanced at a lightening pace, the ability for open interaction in complex domains remains extremely constrained. 
This thesis advances textual-based DIS by introducing a new architecture that allows a seemingly infinite number of complex and branching storylines to be pursued by the user in any domain they choose. The approach uses case-based reasoning methods that leverages stories describing real world events and activities that ordinary people publish to their weblogs every day. The base generation algorithm uses information retrieval techniques to find similar sentences in a corpus of over 1.5 million stories. The weblog stories containing these sentences are used as a proxy for the user’s unfolding composition and the next sentence from the identified weblog story is used as the system’s contribution. To further improve the quality of computer responses, the candidates are reranked using a richer set of linguistic features and finally, portions of the text are adapted to better conform to the discourse of the user’s story. Each of these components is evaluated by crowd-sourcing hundreds of users, thousands of generated stories and tens of thousands of user ratings. 
Moving to this type of data-driven, case-based architecture allows the narrative variation in DIS to scale massively, only limited by the number of stories that can be collected from the Web. The breadth and depth of human experiences captured by this approach pushes us closer to free narrative interaction in the full breadth of complex social domains.

http://reidswanson.com/papers/SwansonThesis.pdf

Enabling open domain interactive storytelling using a data-driven case-based approach

Free-text interactive fiction allows players to narrate the actions of protagonists via natural language input, which are automatically directed to appropriate storyline outcomes using natural language processing techniques. We describe an authoring platform called the Data-driven Interactive Narrative Engine (DINE), which supports free-text interactive fiction by connecting player input to authored outcomes using unsupervised text classification techniques based on text corpus statistics. We hypothesize that the coherence of the interaction, as judged by the players of a DINE scenario, is dependent on specific design choices made by the author. We describe three empirical experiments with crowdsourced subjects to investigate how authoring choices impacted the coherence of the interaction, finding that scenario design and writing style can predict significant differences.

Effective Scenario Designs for Free-Text Interactive Fiction

Automated text classification technologies have enabled researchers to amass enormous collections of personal narratives posted to English-language weblogs. In this paper, we explore analogous approaches to identify personal narratives in Chinese weblog posts as a precursor to the future empirical studies of cross-cultural differences in narrative structure. We describe the collection of over half a million posts from a popular Chinese weblog hosting service, and the manual annotation of story and nonstory content in sampled posts. Using supervised machine learning methods, we developed an automated text classifier for personal narratives in Chinese posts, achieving classification accuracy comparable to previous work in English. Using this classifier, we automatically identify over sixty-four thousand personal narratives for use in future cross-cultural analyses and Chinese-language applications of narrative corpora.

/pdf/identifying-personal-narratives-in-chinese-weblog-posts-1grkm9ss58.pdf

Identifying Personal Narratives in Chinese Weblog Posts

People naturally anthropomorphize the movement of nonliving objects, as social psychologists Fritz Heider and Marianne Simmel demonstrated in their influential 1944 research study. When they asked participants to narrate an animated film of two triangles and a circle moving in and around a box, participants described the shapes' movement in terms of human actions. Using a framework for authoring and annotating animations in the style of Heider and Simmel, we established new crowdsourced datasets where the motion trajectories of animated shapes are labeled according to the actions they depict. We applied two machine learning approaches, a spatial-temporal bag-of-words model and a recurrent neural network, to the task of automatically recognizing actions in these datasets. Our best results outperformed a majority baseline and showed similarity to human performance, which encourages further use of these datasets for modeling perception from motion trajectories. Future progress on simulating human-like motion perception will require models that integrate motion information with top-down contextual knowledge.

/pdf/recognizing-human-actions-in-the-motion-trajectories-of-50fbfbbdtm.pdf

Recognizing Human Actions in the Motion Trajectories of Shapes

Current interactive entertainment is not strategy-aware, in that it does not allow users to execute a wide range of real-world strategies in the context of the artificial environment, it does not recognize when users are trying to execute these strategies, and it does not include software opponents and partners that are capable of executing a varied range of strategies in response to user actions. A solution to this problem is to explicitly identify a broad range of real-world strategies that people would use if the environment of the interactive entertainment were real rather than fiction, to represent these strategies to identify their component features, and to design the entertainment environment so that these component features are supported and recognized. This paper gives an example of how the first two steps of this solution can be done in the political domain by examining 60 explicit strategies outlined in Machiavelli’s The Prince.

/pdf/playing-chess-with-machiavelli-improving-interactive-30jl7byoyk.pdf

Andrew S. Gordon

Papers

Enabling open domain interactive storytelling using a data-driven case-based approach

Effective Scenario Designs for Free-Text Interactive Fiction

Identifying Personal Narratives in Chinese Weblog Posts

Recognizing Human Actions in the Motion Trajectories of Shapes

Playing Chess with Machiavelli: Improving Interactive Entertainment with Explicit Strategies