Representation and learning of commonsense knowledge is one of the foundational problems in the quest to enable deep language understanding. This issue is particularly challenging for understanding casual and correlational relationships between events. While this topic has received a lot of interest in the NLP community, research has been hindered by the lack of a proper evaluation framework. This paper attempts to address this problem with a new framework for evaluating story understanding and script learning: the `Story Cloze Test’. This test requires a system to choose the correct ending to a four-sentence story. We created a new corpus of 50k five-sentence commonsense stories, ROCStories, to enable this evaluation. This corpus is unique in two ways: (1) it captures a rich set of causal and temporal commonsense relations between daily events, and (2) it is a high quality collection of everyday life stories that can also be used for story generation. Experimental evaluation shows that a host of baselines and state-of-the-art models based on shallow language understanding struggle to achieve a high score on the Story Cloze Test. We discuss these implications for script and story learning, and offer suggestions for deeper language understanding.

/pdf/a-corpus-and-cloze-evaluation-for-deeper-understanding-of-4ihd6svu8v.pdf

A Corpus and Cloze Evaluation for Deeper Understanding of Commonsense Stories

From Discourse to Logic

In this paper we give an overview of the Knowledge Base Population (KBP) track at TAC 2010. The main goal of KBP is to promote research in discovering facts about entities and expanding a structured knowledge base with this information. A large source collection of newswire and web documents is provided for systems to discover information. Attributes (a.k.a. “slots”) derived from Wikipedia infoboxes are used to create the reference knowledge base (KB). KBP2010 includes the following four tasks: (1) Regular Entity Linking, where names must be aligned to entities in the KB; (2) Optional Entity linking, without using Wikipedia texts; (3) Regular Slot Filling, which requires a system to automatically discover the attributes of specified entities from the source document collection and use them to expand the KB; (4) Surprise Slot Filling, which requires a system to return answers regarding new slot types within a short time period. KBP2010 has attracted many participants (over 45 teams registered for KBP 2010 (not including the RTEKBP Validation Pilot task), among which 23 teams submitted results). In this paper we provide an overview of the task definition and annotation challenges associated with KBP2010. Then we summarize the evaluation results and discuss the lessons that we have learned based on detailed analysis.

Overview of the TAC 2010 Knowledge Base Population Track

Evaluating temporal relations in clinical text: 2012 i2b2 Challenge.

We describe SUTIME, a temporal tagger for recognizing and normalizing temporal expressions in English text. SUTIME is available as part of the Stanford CoreNLP pipeline and can be used to annotate documents with temporal information. It is a deterministic rule-based system designed for extensibility. Testing on the TempEval-2 evaluation corpus shows that this system outperforms state-of-the-art techniques.

/pdf/sutime-a-library-for-recognizing-and-normalizing-time-53w0y4a0ke.pdf

SUTime: A library for recognizing and normalizing time expressions

In this paper we provide a description of TimeML, a rich specification language for event and temporal expressions in natural language text, developed in the context of the AQUAINT program on Question Answering Systems. Unlik em ost previous work on event annotation, TimeML capture st hree distinct phenomena in temporal markup: (1) it systematically anchors event predicates to a broad range of temporally denotating expressions; (2) it orders event expressions in text relative to one another, both intrasententially and in discourse; and (3) it allows for a delayed (underspecified) interpretation of partially determined temporal expressions. We demonstrate the expressiveness of TimeML for a broad range of syntactic and semantic contexts, including aspectual predication, modal subordination, and an initial treatment of lexical and constructional causation in text.

/pdf/timeml-robust-specification-of-event-and-temporal-1itu2tdql0.pdf

TimeML: Robust Specification of Event and Temporal Expressions in Text

The TempEval task proposes a simple way to evaluate automatic extraction of temporal relations. It avoids the pitfalls of evaluating a graph of inter-related labels by defining three sub tasks that allow pairwise evaluation of temporal relations. The task not only allows straightforward evaluation, it also avoids the complexities of full temporal parsing.

/pdf/semeval-2007-task-15-tempeval-temporal-relation-3nzup1z417.pdf

SemEval-2007 Task 15: TempEval Temporal Relation Identification

In this paper we provide a description of TimeML, a rich specification language for event and temporal expressions in natural language text, developed in the context of the AQUAINT program on Question Answering Systems. Unlike most previous work on event annotation, TimeML captures three distinct phenomena in temporal markup: (1) it systematically anchors event predicates to a broad range of temporally denotating expressions; (2) it orders event expressions in text relative to one another, both intrasententially and in discourse; and (3) it allows for a delayed (underspecified) interpretation of partially determined temporal expressions. We demonstrate the expressiveness of TimeML for a broad range of syntactic and semantic contexts, including aspectual predication, modal subordination, and an initial treatment of lexical and constructional causation in text.

https://xml.coverpages.org/TimeML-SpecLang200401.pdf

The Specification Language TimeML.

Making use of latent semantic analysis, we explore the hypothesis that local linguistic context can serve to identify multi-word expressions that have non-compositional meanings. We propose that vector-similarity between distribution vectors associated with an MWE as a whole and those associated with its constituent parts can serve as a good measure of the degree to which the MWE is compositional. We present experiments that show that low (cosine) similarity does, in fact, correlate with non-compositionality.

/pdf/automatic-identification-of-non-compositional-multi-word-b7hx0x69ts.pdf

Automatic Identification of Non-Compositional Multi-Word Expressions using Latent Semantic Analysis

The main focus of the seminar was on TimeML-based temporal annotation
and reasoning. We were concerned with three main points: 
determining how effectively one can use the TimeML language for
consistent annotation, determining how useful such annotation is for
further processing, and determining what modifications should be applied
to the standard to improve its usefulness in applications such as
question-answering and information retrieval.

/pdf/05151-summary-annotating-extracting-and-reasoning-about-time-39f3131v9u.pdf

Graham Katz

Papers

TimeML: Robust Specification of Event and Temporal Expressions in Text

SemEval-2007 Task 15: TempEval Temporal Relation Identification

The Specification Language TimeML.

Automatic Identification of Non-Compositional Multi-Word Expressions using Latent Semantic Analysis

05151 Summary—Annotating, Extracting and Reasoning about Time and Events