Data Mining Practical Machine Learning Tools and Techniques

We present a data-driven approach to generating responses to Twitter status posts, based on phrase-based Statistical Machine Translation. We find that mapping conversational stimuli onto responses is more difficult than translating between languages, due to the wider range of possible responses, the larger fraction of unaligned words/phrases, and the presence of large phrase pairs whose alignment cannot be further decomposed. After addressing these challenges, we compare approaches based on SMT and Information Retrieval in a human evaluation. We show that SMT outperforms IR on this task, and its output is preferred over actual human responses in 15% of cases. As far as we are aware, this is the first work to investigate the use of phrase-based SMT to directly translate a linguistic stimulus into an appropriate response.

/pdf/data-driven-response-generation-in-social-media-4a755ege4w.pdf

Data-Driven Response Generation in Social Media

Representation and learning of commonsense knowledge is one of the foundational problems in the quest to enable deep language understanding. This issue is particularly challenging for understanding casual and correlational relationships between events. While this topic has received a lot of interest in the NLP community, research has been hindered by the lack of a proper evaluation framework. This paper attempts to address this problem with a new framework for evaluating story understanding and script learning: the `Story Cloze Test’. This test requires a system to choose the correct ending to a four-sentence story. We created a new corpus of 50k five-sentence commonsense stories, ROCStories, to enable this evaluation. This corpus is unique in two ways: (1) it captures a rich set of causal and temporal commonsense relations between daily events, and (2) it is a high quality collection of everyday life stories that can also be used for story generation. Experimental evaluation shows that a host of baselines and state-of-the-art models based on shallow language understanding struggle to achieve a high score on the Story Cloze Test. We discuss these implications for script and story learning, and offer suggestions for deeper language understanding.

/pdf/a-corpus-and-cloze-evaluation-for-deeper-understanding-of-4ihd6svu8v.pdf

A Corpus and Cloze Evaluation for Deeper Understanding of Commonsense Stories

Research in open-domain commonsense reasoning has been hindered by the lack of evaluation metrics for judging progress and comparing alternative approaches. Taking inspiration from large-scale question sets used in natural language processing research, we authored one thousand English-language questions that directly assess commonsense causal reasoning, called the Choice Of Plausible Alternatives (COPA) evaluation. Using a forced-choice format, each question gives a premise and two plausible causes or effects, where the correct choice is the alternative that is more plausible than the other. This paper describes the authoring methodology that we used to develop a validated question set with sufficient breadth to advance open-domain commonsense reasoning research. We discuss the design decisions made during the authoring process, and explain how these decisions will affect the design of high-scoring systems. We also present the performance of multiple baseline approaches that use statistical natural language processing techniques, establishing initial benchmarks for future systems.

/pdf/choice-of-plausible-alternatives-an-evaluation-of-542w8lcuze.pdf

Choice of Plausible Alternatives: An Evaluation of Commonsense Causal Reasoning

SemEval-2012 Task 7 presented a deceptively simple challenge: given an English sentence as a premise, select the sentence amongst two alternatives that more plausibly has a causal relation to the premise. In this paper, we describe the development of this task and its motivation. We describe the two systems that competed in this task as part of SemEval-2012, and compare their results to those achieved in previously published research. We discuss the characteristics that make this task so difficult, and offer our thoughts on how progress can be made in the future.

/pdf/semeval-2012-task-7-choice-of-plausible-alternatives-an-4ggv4b9b29.pdf

SemEval-2012 Task 7: Choice of Plausible Alternatives: An Evaluation of Commonsense Causal Reasoning

Stories of people's everyday experiences have long been the focus of psychology and sociology research, and are increasingly being used in innovative knowledge-based technologies. However, continued research in this area is hindered by the lack of standard corpora of sufficient size and by the costs of creating one from scratch. In this paper, we describe our efforts to develop a standard corpus for researchers in this area by identifying personal stories in the tens of millions of blog posts in the ICWSM 2009 Spinn3r Dataset. Our approach was to employ statistical text classification technology on the content of blog entries, which required the creation of a sufficiently large set of annotated training examples. We describe the development and evaluation of this classification technology and how it was applied to the dataset in order to identify nearly a million

/pdf/identifying-personal-stories-in-millions-of-weblog-entries-oo9zkh3fiq.pdf

Identifying Personal Stories in Millions of Weblog Entries

Interactive storytelling is an interesting cross-disciplinary area that has importance in research as well as entertainment In this paper we explore a new area of interactive storytelling that blurs the line between traditional interactive fiction and collaborative writing We present a system where the user and computer take turns in writing sentences of a fictional narrative Sentences contributed by the computer are selected from a collection of millions of stories extracted from Internet weblogs By leveraging the large amounts of personal narrative content available on the web, we show that even with a simple approach our system can produce compelling stories with our users

/pdf/say-anything-a-massively-collaborative-open-domain-story-5802flkrkb.pdf

Say Anything: A Massively Collaborative Open Domain Story Writing Companion

One of the central problems in building broad-coverage story understanding systems is generating expectations about event sequences, i.e. predicting what happens next given some arbitrary narrative context. In this paper, we describe how a large corpus of stories extracted from Internet weblogs was used to learn a probabilistic model of event sequences using statistical language modeling techniques. Our approach was to encode weblog stories as sequences of events, one per sentence in the story, where each event was represented as a pair of descriptive key words extracted from the sentence. We then applied statistical language modeling techniques to each of the event sequences in the corpus. We evaluated the utility of the resulting model for the tasks of narrative event ordering and event prediction.

/pdf/learning-a-probabilistic-model-of-event-sequences-from-ldo2bz5olm.pdf

Learning a Probabilistic Model of Event Sequences From Internet Weblog Stories

: Large corpora of parsed sentences with semantic role labels (e.g. PropBank) provide training data for use in the creation of high-performance automatic semantic role labeling systems. Despite the size of these corpora, individual verbs (or rolesets) often have only a handful of instances in these corpora, and only a fraction of English verbs have even a single annotation. In this paper, we describe an approach for dealing with this sparse data problem, enabling accurate semantic role labeling for novel verbs (rolesets) with only a single training example. Our approach involves the identification of syntactically similar verbs found in Prop-Bank, the alignment of arguments in their corresponding rolesets, and the use of their corresponding annotations in Prop-Bank as surrogate training data.

https://ict.usc.edu/pubs/Generalizing%20Semantic%20Role%20Annotations%20Across%20Syntactically%20Similar%20Verbs.pdf

Generalizing semantic role annotations across syntactically similar verbs

Moral Foundations Theory (MFT) and the Model of Intuitive Morality and Exemplars (MIME) contend that moral judgments are built on a universal set of basic moral intuitions. A large body of research...

Reid Swanson

Papers

Identifying Personal Stories in Millions of Weblog Entries

Say Anything: A Massively Collaborative Open Domain Story Writing Companion

Learning a Probabilistic Model of Event Sequences From Internet Weblog Stories

Generalizing semantic role annotations across syntactically similar verbs

Extracting Latent Moral Information from Text Narratives: Relevance, Challenges, and Solutions