Expanding the scope of the ATIS task: the ATIS-3 corpus
Deborah A. Dahl,Madeleine Bates,Michael Brown,William Fisher,Kate Hunicke-Smith,David Pallett,Christine Pao,Alexander I. Rudnicky,Elizabeth Shriberg +8 more
- pp 43-48
Reads0
Chats0
TLDR
The migration of the ATIS task to a richer relational database and development corpus (ATIS-3) and the ATis-3 corpus is described, including breakdowns of data by type (e.g. context-independent, context-dependent, and unevaluable) and variations in the data collected at different sites.Abstract:
The Air Travel Information System (ATIS) domain serves as the common evaluation task for ARPA spoken language system developers. To support this task, the Multi-Site ATIS Data COllection Working group (MADCOW) coordinates data collection activities. This paper describes recent MADCOW activities. In particular, this paper describes the migration of the ATIS task to a richer relational database and development corpus (ATIS-3) and describes the ATIS-3 corpus. The expanded database, which includes information on 46 US and Canadian cities and 23,457 flights, was released in the fall of 1992, and data collection for the ATIS-3 corpus began shortly thereafter. The ATIS-3 corpus now consists of a total of 8297 released training utterances and 3211 utterances reserved for testing, collected at BBN, CMU, MIT, NIST and SRI. 2906 of the training utterances have been annotated with the correct information from the database. This paper describes the ATIS-3 corpus in detail, including breakdowns of data by type (e.g. context-independent, context-dependent, and unevaluable)and variations in the data collected at different sites. This paper also includes a description of the ATIS-3 database. Finally, we discuss future data collection and evaluation plans.read more
Citations
More filters
Proceedings ArticleDOI
QuAC: Question Answering in Context
Eunsol Choi,He He,Mohit Iyyer,Mohit Iyyer,Mark Yatskar,Wen-tau Yih,Yejin Choi,Yejin Choi,Percy Liang,Luke Zettlemoyer +9 more
TL;DR: QuAC introduces challenges not found in existing machine comprehension datasets: its questions are often more open-ended, unanswerable, or only meaningful within the dialog context, as it shows in a detailed qualitative evaluation.
Posted Content
Learning End-to-End Goal-Oriented Dialog
TL;DR: In this article, an end-to-end dialog system based on memory networks is proposed for goal-oriented reservation systems, which can reach promising, yet imperfect, performance and learn to perform non-trivial operations.
Proceedings Article
Online Learning of Relaxed CCG Grammars for Parsing to Logical Form
Luke Zettlemoyer,Michael Collins +1 more
TL;DR: A key idea is to introduce non-standard CCG combinators that relax certain parts of the grammar—for example allowing flexible word order, or insertion of lexical items— with learned costs.
Posted Content
Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Task
Tao Yu,Rui Zhang,Kai Yang,Michihiro Yasunaga,Dongxu Wang,Zifan Li,James Ma,Irene Li,Qingning Yao,Shanelle Roman,Zilin Zhang,Dragomir R. Radev +11 more
TL;DR: This work defines a new complex and cross-domain semantic parsing and text-to-SQL task so that different complicated SQL queries and databases appear in train and test sets and experiments with various state-of-the-art models show that Spider presents a strong challenge for future research.
Book
Evaluating Natural Language Processing Systems: An Analysis and Review
TL;DR: This comprehensive state-of-the-art book is the first devoted to the important and timely issue of evaluating NLP systems, and provides a wide-ranging and careful analysis of evaluation concepts, reinforced with extensive illustrations.
References
More filters
Proceedings ArticleDOI
The ATIS spoken language systems pilot corpus
TL;DR: This pilot marks the first full-scale attempt to collect a corpus to measure progress in Spoken Language Systems that include both a speech and natural language component and provides guidelines for future efforts.
Proceedings ArticleDOI
Evaluation of spoken language systems: the ATIS domain
TL;DR: This paper will address the emerging standards for evaluation of spoken language systems with quantifiable measures essential for comparing results and assessing claims.
Proceedings ArticleDOI
Multi-site data collection and evaluation in spoken language understanding
Lynette Hirschman,M. Bates,Deborah A. Dahl,W. Fisher,J. Garofolo,D. Pallett,K. Hunicke-Smith,Patti Price,Alexander I. Rudnicky,Evelyne Tzoukermann +9 more
TL;DR: This work focuses here on selection of training and test data, evaluation of language understanding, and the continuing search for evaluation methods that will correlate well with expected performance of the technology in applications.
Proceedings ArticleDOI
Multi-site data collection for a spoken language corpus
TL;DR: A recently collected spoken language corpus for the ATIS (Air Travel Information System) domain is described and the motivation for this effort, the goals, the implementation of a multi-site data collection paradigm, and the accomplishments of MADCOW are summarized.
Related Papers (5)
Learning to map sentences to logical form: structured classification with probabilistic categorial grammars
Luke Zettlemoyer,Michael Collins +1 more