Wizard of Oz studies — why and how
01 Dec 1993-Knowledge Based Systems (Elsevier Science Publishers B. V.PUB568Amsterdam, The Netherlands, The Netherlands)-
TL;DR: The focus of the work described in the paper is on the practical execution of the studies and the methodological conclusions drawn on the basis of the authors' experience, and the methods used and the conclusions drawn are of relevance also to other kinds of intelligent interfaces.
Abstract: Current approaches to the development of natural language dialogue systems are discussed, and it is claimed that they do not sufficiently consider the unique qualities of man-machine interaction as...
Citations
More filters
Book•
28 Apr 2009TL;DR: In this paper, the authors provide an overview and instruction regarding the evaluation of interactive information retrieval systems with users and present core instruments and data collection techniques and measures, as well as a discussion of outstanding challenges and future research directions.
Abstract: This paper provides overview and instruction regarding the evaluation of interactive information retrieval systems with users The primary goal of this article is to catalog and compile material related to this topic into a single source This article (1) provides historical background on the development of user-centered approaches to the evaluation of interactive information retrieval systems; (2) describes the major components of interactive information retrieval system evaluation; (3) describes different experimental designs and sampling strategies; (4) presents core instruments and data collection techniques and measures; (5) explains basic data analysis techniques; and (4) reviews and discusses previous studies This article also discusses validity and reliability issues with respect to both measures and methods, presents background information on research ethics and discusses some ethical issues which are specific to studies of interactive information retrieval (IIR) Finally, this article concludes with a discussion of outstanding challenges and future research directions
565 citations
28 Jul 2012
TL;DR: This work systematically reviews how researchers conducted Wizard of Oz experiments published in the primary HRI publication venues from 2001 -- 2011 and proposes new reporting guidelines to aid future research.
Abstract: Many researchers use Wizard of Oz (WoZ) as an experimental technique, but there are methodological concerns over its use, and no comprehensive criteria on how to best employ it. We systematically review 54 WoZ experiments published in the primary HRI publication venues from 2001 -- 2011. Using criteria proposed by Fraser and Gilbert (1991), Green et al. (2004), Steinfeld et al. (2009), and Kelley (1984), we analyzed how researchers conducted HRI WoZ experiments. Researchers mainly used WoZ for verbal (72.2%) and non-verbal (48.1%) processing. Most constrained wizard production (90.7%), but few constrained wizard recognition (11%). Few reported measuring wizard error (3.7%), and few reported pre-experiment wizard training (5.4%). Few reported using WoZ in an iterative manner (24.1%). Based on these results we propose new reporting guidelines to aid future research.
456 citations
01 Mar 2007
441 citations
01 May 2015-International Journal of Human-computer Studies \/ International Journal of Man-machine Studies
TL;DR: Qualitative assessment of the direction of quantitative effects demonstrated that robots were more persuasive and perceived more positively when physically present in a user?s environment than when digitally-displayed on a screen either as a video feed of the same robot or as a virtual character analog.
Abstract: The effects of physical embodiment and physical presence were explored through a survey of 33 experimental works comparing how people interacted with physical robots and virtual agents. A qualitative assessment of the direction of quantitative effects demonstrated that robots were more persuasive and perceived more positively when physically present in a user?s environment than when digitally-displayed on a screen either as a video feed of the same robot or as a virtual character analog; robots also led to better user performance when they were collocated as opposed to shown via video on a screen. However, participants did not respond differently to physical robots and virtual agents when both were displayed digitally on a screen - suggesting that physical presence, rather than physical embodiment, characterizes people?s responses to social robots. Implications for understanding psychological response to physical and virtual agents and for methodological design are discussed. Survey identified 33 works exploring user responses to physical robots and virtual agents.Robot agents had greater influence when physically present than telepresent.No differences were found between physical robots displayed on a screen and virtual agents that looked similar.Physical presence, but not physical embodiment alone, resulted in more favorable responses from participants.
389 citations
TL;DR: This article presents a series of studies that quantitatively demonstrate that simple sensors can support the construction of models that estimate human interruptibility as well as people do, and therefore their use in everyday office environments is both practical and affordable.
Abstract: A person seeking another person's attention is normally able to quickly assess how interruptible the other person currently is. Such assessments allow behavior that we consider natural, socially appropriate, or simply polite. This is in sharp contrast to current computer and communication systems, which are largely unaware of the social situations surrounding their usage and the impact that their actions have on these situations. If systems could model human interruptibility, they could use this information to negotiate interruptions at appropriate times, thus improving human computer interaction.This article presents a series of studies that quantitatively demonstrate that simple sensors can support the construction of models that estimate human interruptibility as well as people do. These models can be constructed without using complex sensors, such as vision-based techniques, and therefore their use in everyday office environments is both practical and affordable. Although currently based on a demographically limited sample, our results indicate a substantial opportunity for future research to validate these results over larger groups of office workers. Our results also motivate the development of systems that use these models to negotiate interruptions at socially appropriate times.
366 citations
Cites methods from "Wizard of Oz studies — why and how"
...This section presents sensors simulated using a Wizard of Oz technique [Dahlb¨ ack et al. 1993; Maulsby et al. 1993]....
[...]
References
More filters
TL;DR: This paper found that the 4-year-old adjusted his speech with regard to the changing capacities of different-aged listeners, and the younger the 2-year old, the greater was the observed speech adjustment.
Abstract: SHATZ, MARILYN, and GELMAN, ROCHEL. The Development of Communication Skills: Modifications in the Speech of Young Children as a Function of Listener. Monographs of the Society for Research in Child Development, 1973, 38 (5, Serial No. 152). This Monograph reports three studies of the 4-year-old's ability to adjust to a listener. In the first study (Study A) 16 Ss were pretested on modified versions of standard tests of "egocentrism." Following these, the children were asked first to tell an adult about a toy and then tell a 2-year-old about that toy. The eight Ss who had 2-year-old siblings were run on the toy task twice: once in an adult-sibling session, and once in an adult-nonsibling session. Finally, tapes were made of spontaneous conversations between the Ss and their mothers. As expected, the Ss performed poorly on the tests of "egocentrism." In contrast, Ss adjusted their speech production to their different listeners. Speech to 2-year-olds contained more short, simple utterances and more attentional utterances. The younger the 2-year-old, the greater was the observed speech adjustment. All children adjusted their speech whether or not they had younger siblings. In Study B tapes of uncontrolled conversations of five 4-year-olds each talking to a 2-year-old and an adult were obtained. Analyses of speech adjustments revealed a pattern of results like those of Study A. In Study C tapes of eight 4-year-olds talking to peers were collected and compared with the taped conversations with their mothers obtained in the first study. Analyses of the peer-directed versus adult-directed speech showed that, with respect to utterance length, the use of various constructions, and attentional utterances, peers were treated like adults. These results, combined with those of Study A, indicate that the 4-year-old adjusts his speech with regard to the changing capacities of different-aged listeners. The results of these studies are discussed with regard to previous work on the preschooler's communication skills and the variety of listener-produced cues that may influence the 4-year-old's tendency to "talk down." The implications of speaker-listener interaction for the process of language acquisition are considered. This content downloaded from 157.55.39.163 on Wed, 21 Sep 2016 04:38:28 UTC All use subject to http://about.jstor.org/terms
834 citations
TL;DR: The results throw doubt on the suggestion that the Late Closure strategy is favoured (in English) mainly because it is efficient in information processing terms and suggest that different languages make use of parsing strategies in an essentially arbitrary way.
441 citations
TL;DR: The “Wizard of Oz” technique for simulating future interactive technology and a partial taxonomy of such simulations is reviewed and a general Wizard of Oz methodology is suggested.
425 citations
Book•
01 Jan 1977
TL;DR: The ability to represent focus of attention in a languageunderstanding system results in a new approach to an important problemin discourse comprehension -- the identification of the referents ofdefinite noun phrases.
Abstract: : This report develops a representation of focus of attention thatcircumscribes discourse contexts within a general representation ofknowledge. Focus of attention is essential to any comprehension processbecause what and how a person understands is strongly influenced bywhere his attention is directed at a given moment. To formalize thenotion of focus, the need for and the use of focus mechanisms areconsidered from the standpoint of building a computer system that canparticipate in a natural language dialogue with a ser, Two ranges offocus, global and immediate, are investigated, and representations forincorporating them in a computer system are developed.The global focus in which an utterance is interpreted is determinedby the total discourse and situational setting of the utterance. Itinfluences what is talked about, how different concepts are introduced,and how concepts are referenced. To encode global focuscomputationally, a representation is developed that highlights thoseitems that are relevant at a given place in a dialogue. The underlyingknowledge representation is segmented into subunits, called focusspaces, that contain those items that are in the focus of attention of adialogue participant during a particular part of the dialogue.Mechanisms are required for updating the focus representation,because, as a dialogue progresses, the objects and actions that arerelevant to the conversation, and therefore in the participants' focusof attention, change. Procedures are described for deciding when andhow to shift focus in task-oriented dialogues, i.e., in dialogues inwhich the participants are cooperating in a shared task. Theseprocedures are guided by a representation of the task being performed.The ability to represent focus of attention in a languageunderstanding system results in a new approach to an important problemin discourse comprehension -- the identification of the referents ofdefinite noun phrases.
410 citations