scispace - formally typeset
Search or ask a question
Proceedings ArticleDOI

User studies and the design of natural language systems

10 Apr 1989-pp 116-123
TL;DR: It is concluded that previous approaches have neglected to evaluate systems in the context of their use, e.g. solving a task requiring data retrieval, and how these might be satisfied by future Natural Language systems is discussed.
Abstract: This paper presents a critical discussion of the various approaches that have been used in the evaluation of Natural Language systems. We conclude that previous approaches have neglected to evaluate systems in the context of their use, e.g. solving a task requiring data retrieval. This raises questions about the validity of such approaches. In the second half of the paper, we report a laboratory study using the Wizard of Oz technique to identify NL requirements for carrying out this task. We evaluate the demands that task dialogues collected using this technique, place upon a prototype Natural Language system. We identify three important requirements which arose from the task that we gave our subjects: operators specific to the task of database access, complex contextual reference and reference to the structure of the information source. We discuss how these might be satisfied by future Natural Language systems.

Content maybe subject to copyright    Report

Citations
More filters
Posted Content
TL;DR: Natural language interfaces to databases (NLIDBs) as discussed by the authors have been studied extensively in the field of natural language processing and have attracted much attention in the last few decades, especially for query languages, form-based interfaces and graphical interfaces.
Abstract: This paper is an introduction to natural language interfaces to databases (NLIDBs). A brief overview of the history of NLIDBs is first given. Some advantages and disadvantages of NLIDBs are then discussed, comparing NLIDBs to formal query languages, form-based interfaces, and graphical interfaces. An introduction to some of the linguistic problems NLIDBs have to confront follows, for the benefit of readers less familiar with computational linguistics. The discussion then moves on to NLIDB architectures, portability issues, restricted natural language input systems (including menu-based NLIDBs), and NLIDBs with reasoning capabilities. Some less explored areas of NLIDB research are then presented, namely database updates, meta-knowledge questions, temporal questions, and multi-modal NLIDBs. The paper ends with reflections on the current state of the art.

694 citations

Journal ArticleDOI
TL;DR: This paper is an introduction to natural language interfaces to databases (NLIDBS) and some less explored areas of NLIDB research are presented, namely database updates, meta-knowledge questions, temporal questions, and multi-modal NLIDBS.
Abstract: This paper is an introduction to natural language interfaces to databases (NLIDBS). A brief overview of the history of NLIDBS is first given. Some advantages and disadvantages of NLIDBS are then discussed, comparing NLIDBS to formal query languages, form-based interfaces, and graphical interfaces. An introduction to some of the linguistic problems NLIDBS have to confront follows, for the benefit of readers less familiar with computational linguistics. The discussion then moves on to NLIDB architectures, portability issues, restricted natural language input systems (including menu-based NLIDBS), and NLIDBS with reasoning capabilities. Some less explored areas of NLIDB research are then presented, namely database updates, meta-knowledge questions, temporal questions, and multi-modal NLIDBS. The paper ends with reflections on the current state of the art.

679 citations


Cites methods from "User studies and the design of natu..."

  • ...y tables had to be combined, and in queries that were not similar to the ones the users had encountered duringthe training period. Information about similar experiments can be found in [62] and [86]. [103] discusses various approaches that have been used in the evaluation of natural language systems, and describes an experiment where first a Wizard of Oz was used to collect sample user questions, and th...

    [...]

Journal ArticleDOI
TL;DR: The “Wizard of Oz” technique for simulating future interactive technology and a partial taxonomy of such simulations is reviewed and a general Wizard of Oz methodology is suggested.

425 citations

Dissertation
01 Jan 1994
TL;DR: Experimental results indicate that Newt can be personalized to serve some of the news ltering needs of the user, in particular, those that are more regular and predictable, and the basic framework for personalized information ltering agents is presented.
Abstract: A personalized information ltering system must specialize to current interests of the user and adapt as they change over time. It must also explore newer domains for potentially interesting information. A learning approach to building personalized information ltering systems is proposed. The system is designed as a collection of information ltering interface agents. Interface Agents are intelligent and autonomous computer programs which learn users' preferences and act on their behalf | electronic personal assistants that automate tasks for the user. This thesis presents the basic framework for personalized information ltering agents, and describes an implementation, \Newt", built using the framework. Newt uses a keyword based ltering algorithm. The learning mechanisms used are relevance feedback and the genetic algorithm. The user interface is friendly and accessible to both naive as well as power users. Experimental results indicate that Newt can be personalized to serve some of the news ltering needs of the user, in particular, those that are more regular and predictable. Relevance feedback is good for specializing to user interests. The genetic algorithm causes the system to adapt and explore for new types of information. This demonstrates that Interface Agents are a promising approach to the problem of designing personalized information ltering. Abstract A personalized information ltering system must specialize to current interests of the user and adapt as they change over time. It must also explore newer domains for potentially interesting information. A learning approach to building personalized information ltering systems is proposed. The system is designed as a collection of information ltering interface agents. Interface Agents are intelligent and autonomous computer programs which learn users' preferences and act on their behalf | electronic personal assistants that automate tasks for the user. This thesis presents the basic framework for personalized information ltering agents, and describes an implementation, \Newt", built using the framework. Newt uses a keyword based ltering algorithm. The learning mechanisms used are relevance feedback and the genetic algorithm. The user interface is friendly and accessible to both naive as well as power users. Experimental results indicate that Newt can be personalized to serve some of the news ltering needs of the user, in particular, those that are more regular and predictable. Relevance feedback is good for specializing to user interests. The genetic algorithm causes the system to adapt and explore for new types of information. This demonstrates that Interface Agents are a promising approach to the problem …

229 citations


Cites background from "User studies and the design of natu..."

  • ...they want organization above the level of individual objects [46]....

    [...]

Journal ArticleDOI
TL;DR: The paper discusses the issues, both general to language and task-specific, involved in front end design, and the way these have been addressed, concentrating on the work of the last decade, indicating that future progress will depend on the one hand on general advances in natural language processing, and on the other on expanding the capabilities of traditional databases.
Abstract: This paper reviews the current state of the art in natural language access to databases. This has been a long-standing area of work in natural language processing. But though some commercial systems are now available, providing front ends has proved much harder than was expected, and the necessary limitations on front ends have to be recognized. The paper discusses the issues, both general to language and task-specific, involved in front end design, and the way these have been addressed, concentrating on the work of the last decade. The focus is on the central process of translating a natural language question into a database query, but other supporting functions are also covered. The points are illustrated by the use of a single example application. The paper concludes with an evaluation of the current state, indicating that future progress will depend on the one hand on general advances in natural language processing, and on the other on expanding the capabilities of traditional databases.

166 citations

References
More filters
01 Jan 1977

272 citations

Journal ArticleDOI
Lance A. Miller1
TL;DR: The objective in this study was to obtain detailed empirical information about the nature of natural language “programming” to bring to bear on the issue of increasing the usability of computer language interfaces.
Abstract: Our objective in this study was to obtain detailed empirical information about the nature of natural language “programming” to bring to bear on the issueosf increasing the usability of computer language interfaces. Although we expected numerous difficulties to be detected concerning the potentioalf actually implementing a system to interpret natural language programs, we were not prepared for the magnitude of what we see as being the three major obstacles: style, semantics, and world knowledge. Concerning the first, there is little way in which the vast differences in styles could be increased: programming-language style is simply alien to natural specification. With respect to semantics, we also were unprepared to find out the extent to which the selection of the appropriate “meaning” (to a word, phrase, or sentence) is dependent upon the immediate and prior context. And as for world nowledge, we suspect that the extent to which shared experiences and knowledge are critical to procedural communication and understanding among people has barely been hintaetd by our present data.

192 citations

Journal ArticleDOI
P. Reisner1
TL;DR: This paper describes a series of psychological experiments used to test a new data base query language, aiming to make psychological testing of a language part of the design and development process.
Abstract: This paper describes a series of psychological experiments used to test a new data base query language. The intent is to make psychological testing of a language part of the design and development process. By testing a language while it is still under development, features that require changing can bs identified and the changes made.

187 citations

DOI
15 May 1981
TL;DR: It is demonstrated, using protocols of actual interactions with a question-answering system, that users of these systems expect to engage in a conversation whose coherence is manifested in the interpendence of their (often unstated) plans and goals with those of the system.
Abstract: : We demonstrate, using protocols of actual interactions with a question-answering system, that users of these systems expect to engage in a conversation whose coherence is manifested in the interpendence of their (often unstated) plans and goals with those of the system. Since these problems are even more obvious in other forms of natural-language understanding systems, such as task-oriented dialogue systems, techniques for engaging in question-answering conversation should be special cases of general conversational abilities. We characterize dimensions along which language understanding systems might differ and, based partly on this analysis, propose a new system architecture, centered around recognizing the user's plans and planning helpful responses, which can be applied to a number of possible application areas. To illustrate progress to date, we discuss two implemented systems, one operating in a simple question-answering framework, and the other in a decision support framework for which both graphic and linguistic means of communication are available. (Author)

167 citations

Posted Content
TL;DR: The design and results of a field evaluation of a natural language system-NLS-used for data retrieval and its practical usefulness are presented.
Abstract: Although a large number of natural language database interfaceshave been developed, there have been few empirical studies of theirpractical usefulness. This paper presents the design and results of afield evaluation of a natural language system - NLS - used for dataretrieval .A balanced, multifactorial design comparing NLS with a referenceretrieval language, SQL, is described. The data are analyzed on twolevels: work task (n=87) and query (n=1081). SQL performed betterthan NLS on a variety of measures, but NLS required less effort touse. Subjects performed much poorer than expected based on theresults of laboratory studies. This finding is attributed to thecomplexity of the field setting and to optimism in grading laboratoryexperiments.The methodology developed for studying computer languages in realwork settings was successful in consistently measuring differences intreatments over a variety of conditions.

93 citations