TellMeWhy: A Dataset for Answering Why-Questions in Narratives
TL;DR: In this article, the authors introduce a new crowd-sourced dataset that consists of more than 30k questions and free-form answers concerning why characters in short narratives perform the actions described.
Abstract: Answering questions about why characters perform certain actions is central to understanding and reasoning about narratives. Despite recent progress in QA, it is not clear if existing models have the ability to answer "why" questions that may require commonsense knowledge external to the input narrative. In this work, we introduce TellMeWhy, a new crowd-sourced dataset that consists of more than 30k questions and free-form answers concerning why characters in short narratives perform the actions described. For a third of this dataset, the answers are not present within the narrative. Given the limitations of automated evaluation for this task, we also present a systematized human evaluation interface for this dataset. Our evaluation of state-of-the-art models show that they are far below human performance on answering such questions. They are especially worse on questions whose answers are external to the narrative, thus providing a challenge for future QA and narrative understanding research.
Citations
More filters
01 Jan 2022
TL;DR: In this paper , a case study on the French Revolution is presented, based upon Wikidata and Wikipedia, and a prototype helps identifying the first challenges such as dynamic representation and evaluation of a narrative.
Abstract: Humans constantly create narratives to provide explanations for how and why something happens. Designing systems able to build such narratives would therefore contribute to building more human-centric systems, and to support uses like decision-making processes. Here, a narrative is seen as a sequence of events. My thesis investigates how a narrative can be built computationally. Four research questions are identified: representation, construction, link prediction and evaluation. A case study on the French Revolution, based upon Wikidata and Wikipedia is presented. This prototype helps identifying the first challenges such as dynamic representation and evaluation of a narrative.
3 citations
References
More filters
IBM1
TL;DR: This paper proposed a method of automatic machine translation evaluation that is quick, inexpensive, and language-independent, that correlates highly with human evaluation, and that has little marginal cost per run.
Abstract: Human evaluations of machine translation are extensive but expensive. Human evaluations can take months to finish and involve human labor that can not be reused. We propose a method of automatic machine translation evaluation that is quick, inexpensive, and language-independent, that correlates highly with human evaluation, and that has little marginal cost per run. We present this method as an automated understudy to skilled human judges which substitutes for them when there is need for quick or frequent evaluations.
21,126 citations
Proceedings Article•
25 Jul 2004TL;DR: Four different RouGE measures are introduced: ROUGE-N, ROUge-L, R OUGE-W, and ROUAGE-S included in the Rouge summarization evaluation package and their evaluations.
Abstract: ROUGE stands for Recall-Oriented Understudy for Gisting Evaluation. It includes measures to automatically determine the quality of a summary by comparing it to other (ideal) summaries created by humans. The measures count the number of overlapping units such as n-gram, word sequences, and word pairs between the computer-generated summary to be evaluated and the ideal summaries created by humans. This paper introduces four different ROUGE measures: ROUGE-N, ROUGE-L, ROUGE-W, and ROUGE-S included in the ROUGE summarization evaluation package and their evaluations. Three of them have been used in the Document Understanding Conference (DUC) 2004, a large-scale summarization evaluation sponsored by NIST.
9,293 citations
TL;DR: The semantic structure of texts can be described both at the local microlevel and at a more global macrolevel, and a model for text comprehension based on this notion accounts for the formation of a coherent semantic text base in terms of a cyclical process constrained by limitations of working memory.
Abstract: The semantic structure of texts can be described both at the local microlevel and at a more global macrolevel A model for text comprehension based on this notion accounts for the formation of a coherent semantic text base in terms of a cyclical process constrained by limitations of working memory Furthermore, the model includes macro-operators, whose purpose is to reduce the information in a text base to its gist, that is, the theoretical macrostructure These operations are under the control of a schema, which is a theoretical formulation of the comprehender's goals The macroprocesses are predictable only when the control schema can be made explicit On the production side, the model is concerned with the generation of recall and summarization protocols This process is partly reproductive and partly constructive, involving the inverse operation of the macro-operators The model is applied to a paragraph from a psychological research report, and methods for the empirical testing of the model are developed
4,800 citations
01 Oct 2020
TL;DR: Transformers is an open-source library that consists of carefully engineered state-of-the art Transformer architectures under a unified API and a curated collection of pretrained models made by and available for the community.
Abstract: Recent progress in natural language processing has been driven by advances in both model architecture and model pretraining. Transformer architectures have facilitated building higher-capacity models and pretraining has made it possible to effectively utilize this capacity for a wide variety of tasks. Transformers is an open-source library with the goal of opening up these advances to the wider machine learning community. The library consists of carefully engineered state-of-the art Transformer architectures under a unified API. Backing this library is a curated collection of pretrained models made by and available for the community. Transformers is designed to be extensible by researchers, simple for practitioners, and fast and robust in industrial deployments. The library is available at https://github.com/huggingface/transformers.
4,798 citations
Book•
01 Jan 1977TL;DR: Schank and Abelson as mentioned in this paper analyzed the conceptual apparatus necessary to perform even a partial feat of understanding, and their analysis of this apparatus is what is what this book is about.
Abstract: For both people and machines, each in their own way, there is a serious problem in common of making sense out of what they hear, see, or are told about the world. The conceptual apparatus necessary to perform even a partial feat of understanding is formidable and fascinating. Our analysis of this apparatus is what this book is about. Roger C. Schank and Robert P. Abelson from the Introduction (http://www.psypress.com/scripts-plans-goals-and-understanding-9780898591385)
3,163 citations