Prototyping an intelligent agent through Wizard of Oz
Summary (4 min read)
INTRODUCTION
- In particular the set of instructions and commands they adopted.the authors.
- The authors also explore the lessons they learned using the method, which experimenters can apply in their own studies of intelligent interfaces.
- These lessons differ from other Wizard of Oz experiences in being oriented towards prototyping an implementable system, rather than a proof of concept.
- The paper begins with a brief discussion of intelligent agents and the Wizard of Oz method.
Intelligent agents
- When given a goal, [an intelligent agent] could carry out the details of the appropriate computer operations and could ask for and receive advice, offered in human terms, when it was stuck.
- Eager (Cypher, 1991) detects and automates a user’s repetitive actions in HyperCard; it matches examples by parsing text strings and by testing numerical relationships.
- Metamouse (Maulsby, 1989) learns drawing tasks from demonstrations; it applies rules to find significant graphical constraints.
- Unfortunately, because most work on agents stems from the field of Artificial Intelligence, users’ needs are second to algorithm development.
- The traditional approach of system building is an expensive and unlikely way to gain this understanding.
THE TURVY EXPERIMENT
- The authors research concerns both the technical and usability aspects of programming by demonstration.
- For practicality, it must learn under the user’s guidance, so it needs an intuitive and flexible teaching interface.
- The authors decided that an agent metaphor would help us explore the design issues; the agent is Turvy.
- But in practice, Basil frustrated users’ attempts to teach it.
- The authors wanted people to learn through conversation (verbal or graphical) what the agent could understand.
Turvy as agent
- The authors made four key assumptions about the sort of agent they would build, each with consequences for usability.
- Turvy learns from a user’s demonstrations, pointing, and verbal hints.
- It does not have human-level abilities or knowledge; like Metamouse, it forms search and result patterns from low-level features, and users must refer to them during demonstrations.
- Second, unlike Metamouse, Turvy does not equate the user’s demonstration with a procedure; actions may be interpreted as focusing attention or extending a pattern, and Turvy can revise an interpretation as more examples are seen.
Class From User From Turvy
- Third, an implementable Turvy would not have true natural language capabilities, and their system only recognizes spoken or typed keywords and phrases, using an application-specific lexicon.
- Verbal inputs are either commands (like Stop!) or hints about features (like look for a word before a colon), where keywords (word, colon) are compared with actual data at the locus of action to determine their meaning.
- Eager prediction gives users efficient control over learning.
- Turvy has deemed important, without obscuring text or graphic data.
- As a side effect, the users also learn Turvy’s language.
Hypotheses
- Formalizing the inference and interaction models revealed the complexity of various kinds of instructions and the information needed to interpret them.
- This helped us form hypotheses about the way people would construct lessons and the instructions they would use.
- All users would employ the same small set of commands, those given in Table 1, with only minor variations in wording.
- Moreover, if Turvy uttered (perhaps in the form of a question) some instruction the user had previously given but with different wording, users would thereafter adopt Turvy’s wording.
- This hypothesis is based on verbal convergence (Leiser, 1989), as mentioned in the introduction.
Experimental setup
- In their experiment users sat at a Macintosh computer and worked on bibliographic entries using the Microsoft Word text editor.
- Nearby (but out of eye contact) sat Turvy, played by the system designer, who had a second keyboard and mouse also connected to the Macintosh.
- The user would practice on several entries until able to reformat them correctly.
- They spoke more curtly to Turvy than to the facilitator, and referred to Turvy and the Wizard as two separate entities.
- The pre-pilot used a menu, with the facilitator-to-be acting as user.
OBSERVATIONS AND RESULTS
- The authors data consists of video tapes, transcripts, and the experimenters’ subjective observations.
- The authors studied comments made by users while working with Turvy and during interviews.
- The authors also did content analyses, counting the number of bibliographic terms vs. TurvyTalk in users’ instructions, and measuring indicators of confidence, hesitation and confusion at various points of interest during the session.
Command set (Hypothesis 1)
- Users gave a close-fitting subset of the instructions the authors had predicted.
- From Table 1 the authors see that nearly all commands were used and caused no difficulty.
- The actual wordings subjects used were quite consistent, especially after they heard Turvy ask the corresponding question: they would turn it into a command, such as “Do the rest.” (See TurvyTalk, Hypothesis 2).
- Users almost never volunteered vague hints like “I’m repeating actions,” “this is similar,” and even “look here.”.
TurvyTalk (Hypothesis 2)
- The authors found that users did learn to describe things like titles and surnames in terms of their syntax.
- In post-session interviews, all users said that Turvy does not know about bibliographies; few could describe the sort of terminology it does understand, but they could list examples.
- Dividing the entire session into 16 events for different phases of tasks (first example, points where Turvy would err, etc.), the authors counted the number of user utterances referring to features in terms Turvy understood versus those involving bibliographic terminology (eg. “paste after the author’s name”).
- The authors conclude that Turvy’s speech quickly trained users to mirror its language—verbal convergence occurs.
Teaching difficulty (Hypothesis 3)
- One of their chief aims is to make simple tasks easy to teach, and complex tasks teachable with reasonable effort.
- In their study, easy tasks (like changing underlined text to italics) were trivially taught by giving a demonstration.
- All but one user reported that Turvy was easy to teach, once they had realized it learns incrementally and continuously so they needn’t anticipate all special cases.
- One user told us at the outset that no computer could be taught without anticipating all cases, and therefore refused to try.
- The authors found that users had a fairly neutral feeling of control; however, dealing with unexpected cases caused anxiety.
Speech versus pointing (Hypothesis 4)
- One instructional technique the authors hoped to find was pointing to focus attention, but they observed almost none (apart from explicit selections required by tasks).
- When Turvy asked users to explain a new case by “pointing to something in the text,” they were confused, if the distinguishing feature was a property rather than a string.
- The authors concluded that this query is ineffective; instead, Turvy should propose a guess.
Dialog
- The authors found two distinct styles of interaction, not anticipated in their hypotheses.
- Some feel duty-bound to explain expected special cases in advance, but find this hard to do.
- Talkative users say less as they grow more adept at using Turvy; quiet ones stay quiet.
- All users liked the way Turvy is eager to predict after one example, because they believed this gave them more control over learning.
- The results were the same, except that the authors saw some use of pointing (at fields in file listings).
LESSONS LEARNED USING WIZARD OF OZ
- These lessons differ from other Wizard of Oz experiences, being oriented towards implementable systems, rather than proof of concepts.
- The authors did this in Turvy by designing a formal learning model, and by “scripting” the Wizard’s responses by running the tasks through the model and codifying the results.
- A realistic dialog must be constrained by an interaction model that explicitly lists the kinds of instructions the system can understand and the feedback it can formulate.
- By acting as Wizard, facilitator, and interviewer, the experimenters become immersed in the experiment and many important results become obvious.
- Interviews are essential, and video records are useful.
CONCLUSIONS
- The agent, Turvy, learns procedures and data descriptions from one or more examples done by the user, combined with verbal and pointing hints.
- The simulation was constrained by formal models of inference and interaction, so that Turvy would have realistic limitations.
- Moreover, it learned concepts, but not the user’s terminology.
- Turvy because it learns new cases on the fly, and makes good use of both demonstrations and verbal hints.
- The authors learned valuable lessons about the Wizard of Oz, in particular the benefits of formal models, detailed task analysis, and direct feedback from users to the designer.
Did you find this useful? Give us your feedback
Citations
11,929 citations
1,170 citations
1,004 citations
882 citations
Cites background from "Prototyping an intelligent agent th..."
...The robot’s behaviour was partially autonomous and partially remote controlled (Wizard-of-Oz, WoZ technique; see Gould et al. 1983; Dahlback et al. 1993; Maulsby et al. 1993), whereby the illusion is given to the subjects during the experiment that the robot operates fully autonomously....
[...]
650 citations
References
892 citations
"Prototyping an intelligent agent th..." refers background in this paper
...Likewise, Dahlbäck, Jönsson and Ahrenberg (1993) studied differences between human-human and human-computer discourse through a variety of feigned natural language interfaces....
[...]
641 citations
254 citations
242 citations
"Prototyping an intelligent agent th..." refers background in this paper
...A human “Wizard” simulates the system’s intelligence and interacts with the user through a real or mock computer interface....
[...]
206 citations