User studies and the design of natural language systems
Steve Whittaker,Phil Stenton +1 more
- pp 116-123
TLDR
It is concluded that previous approaches have neglected to evaluate systems in the context of their use, e.g. solving a task requiring data retrieval, and how these might be satisfied by future Natural Language systems is discussed.Abstract:
This paper presents a critical discussion of the various approaches that have been used in the evaluation of Natural Language systems. We conclude that previous approaches have neglected to evaluate systems in the context of their use, e.g. solving a task requiring data retrieval. This raises questions about the validity of such approaches. In the second half of the paper, we report a laboratory study using the Wizard of Oz technique to identify NL requirements for carrying out this task. We evaluate the demands that task dialogues collected using this technique, place upon a prototype Natural Language system. We identify three important requirements which arose from the task that we gave our subjects: operators specific to the task of database access, complex contextual reference and reference to the structure of the information source. We discuss how these might be satisfied by future Natural Language systems.read more
Citations
More filters
Proceedings ArticleDOI
A case study of natural language customisation: the practical effects of world knowledge
TL;DR: A field study is reported in which a methodology for the customisation of natural language interfaces to information retrieval applications is tested by customising a commercially available natural language system to a large database of sales and marketing information.
Posted Content
Evaluating Discourse Processing Algorithms
TL;DR: The authors evaluate two different approaches to anaphoric processing in discourse by comparing the accuracy and coverage of two published algorithms for finding the co-specifiers of pronouns in naturally occurring texts and dialogues.
Natural Language For Database Retrieval- A Practical Perspective
Nottingham Polytechnic,Burton St +1 more
TL;DR: An application is highlighted that introduced natural language to an organisation to replace a structured query language and the new users of the natural language system were initially happy with it, but over a period of months began to migrate and preferred the structured querylanguage tool.
Book ChapterDOI
Natural Language For Database Retrieval- A Practical Perspective
TL;DR: An application is highlighted that introduced natural language to an organisation to replace a structured query language and the new users of the natural language system were initially happy with it, but over a period of months began to migrate and preferred the structured querylanguage tool.
References
More filters
Journal ArticleDOI
Natural language programming: styles, strategies, and contrasts
TL;DR: The objective in this study was to obtain detailed empirical information about the nature of natural language “programming” to bring to bear on the issue of increasing the usability of computer language interfaces.
Journal ArticleDOI
Use of Psychological Experimentation as an Aid to Development of a Query Language
TL;DR: This paper describes a series of psychological experiments used to test a new data base query language, aiming to make psychological testing of a language part of the design and development process.
Beyond Question-Answering.
TL;DR: It is demonstrated, using protocols of actual interactions with a question-answering system, that users of these systems expect to engage in a conversation whose coherence is manifested in the interpendence of their (often unstated) plans and goals with those of the system.
Posted Content
A Field Evaluation of Natural Language for Data Retrieval
TL;DR: The design and results of a field evaluation of a natural language system-NLS-used for data retrieval and its practical usefulness are presented.