On Coreference Resolution Performance Metrics
Xiaoqiang Luo
- pp 25-32
TLDR
The paper proposes a Constrained Entity-Alignment F-Measure (CEAF) for evaluating coreference resolution and shows that the best alignment is a maximum bipartite matching problem which can be solved by the Kuhn-Munkres algorithm.Abstract:
The paper proposes a Constrained Entity-Alignment F-Measure (CEAF) for evaluating coreference resolution. The metric is computed by aligning reference and system entities (or coreference chains) with the constraint that a system (reference) entity is aligned with at most one reference (system) entity. We show that the best alignment is a maximum bipartite matching problem which can be solved by the Kuhn-Munkres algorithm. Comparative experiments are conducted to show that the widely-known MUC F-measure has serious flaws in evaluating a coreference system. The proposed metric is also compared with the ACE-Value, the official evaluation metric in the Automatic Content Extraction (ACE) task, and we conclude that the proposed metric possesses some properties such as symmetry and better interpretability missing in the ACE-Value.read more
Citations
More filters
Proceedings Article
CoNLL-2012 Shared Task: Modeling Multilingual Unrestricted Coreference in OntoNotes
TL;DR: The OntoNotes annotation (coreference and other layers) is described and the parameters of the shared task including the format, pre-processing information, evaluation criteria, and presents and discusses the results achieved by the participating systems.
Proceedings Article
Stanford’s Multi-Pass Sieve Coreference Resolution System at the CoNLL-2011 Shared Task
TL;DR: The coreference resolution system submitted by Stanford at the CoNLL-2011 shared task was ranked first in both tracks, with a score of 57.8 in the closed track and 58.3 in the open track.
Journal ArticleDOI
Deterministic coreference resolution based on entity-centric, precision-ranked rules
TL;DR: The two stages of the sieve-based architecture, a mention detection stage that heavily favors recall, followed by coreference sieves that are precision-oriented, offer a powerful way to achieve both high precision and high recall.
Proceedings Article
A Multi-Pass Sieve for Coreference Resolution
Karthik Raghunathan,Heeyoung Lee,Sudarshan Rangarajan,Nate Chambers,Mihai Surdeanu,Dan Jurafsky,Christopher D. Manning +6 more
TL;DR: This work proposes a simple coreference architecture based on a sieve that applies tiers of deterministic coreference models one at a time from highest to lowest precision, and outperforms many state-of-the-art supervised and unsupervised models on several standard corpora.
Proceedings Article
CoNLL-2011 Shared Task: Modeling Unrestricted Coreference in OntoNotes
TL;DR: The CoNLL-2011 shared task involved predicting coreference using OntoNotes data, a new resource that provides multiple integrated annotation layers (parses, semantic roles, word senses, named entities and coreference) that could support joint models.
References
More filters
Journal ArticleDOI
The Hungarian method for the assignment problem
TL;DR: This paper has always been one of my favorite children, combining as it does elements of the duality of linear programming and combinatorial tools from graph theory, and it may be of some interest to tell the story of its origin this article.
Book
Practical Methods of Optimization
TL;DR: The aim of this book is to provide a Discussion of Constrained Optimization and its Applications to Linear Programming and Other Optimization Problems.
Journal ArticleDOI
Algorithms for the Assignment and Transportation Problems
TL;DR: In this paper, algorithms for the solution of the general assignment and transportation problems are presen, and the algorithm is generalized to one for the transportation problem.
Proceedings ArticleDOI
A model-theoretic coreference scoring scheme
TL;DR: This note describes a scoring scheme for the coreference task in MUC6 that improves on the original approach by grounding the scoring scheme in terms of a model; producing more intuitive recall and precision scores; and not requiring explicit computation of the transitive closure of coreference.