Wizard of Oz studies: why and how
01 Feb 1993-Vol. 6, Iss: 4, pp 193-200
TL;DR: It is concluded that empirical studies of the unique qualities of man-machine interaction as distinct from general human discourse are required for the development of user-friendly interactive systems.
Abstract: Current approaches to the development of natural language dialogue systems are discussed, and it is claimed that they do not sufficiently consider the unique qualities of man-machine interaction as distinct from general human discourse. It is concluded that empirical studies of this unique communication situation are required for the development of user-friendly interactive systems. One way of achieving this is through the use of so-called Wizard of Oz studies. The focus of the work described in the paper is on the practical execution of the studies and the methodological conclusions drawn on the basis of the authors' experience. While the focus is on natural language interfaces, the methods used and the conclusions drawn from the results obtained are of relevance also to other kinds of intelligent interfaces.
Citations
More filters
TL;DR: The paper concludes by examining different paradigms regarding ‘social relationships’ of robots and people interacting with them.
Abstract: Social intelligence in robots has a quite recent history in artificial intelligence and robotics. However, it has become increasingly apparent that social and interactive skills are necessary requirements in many application areas and contexts where robots need to interact and collaborate with other robots or humans. Research on human–robot interaction (HRI) poses many challenges regarding the nature of interactivity and ‘social behaviour’ in robot and humans. The first part of this paper addresses dimensions of HRI, discussing requirements on social skills for robots and introducing the conceptual space of HRI studies. In order to illustrate these concepts, two examples of HRI research are presented. First, research is surveyed which investigates the development of a cognitive robot companion. The aim of this work is to develop social rules for robot behaviour (a ‘robotiquette’) that is comfortable and acceptable to humans. Second, robots are discussed as possible educational or therapeutic toys for children with autism. The concept of interactive emergence in human–child interactions is highlighted. Different types of play among children are discussed in the light of their potential investigation in human–robot experiments. The paper concludes by examining different paradigms regarding ‘social relationships’ of robots and people interacting with them.
882 citations
Cites background from "Wizard of Oz studies: why and how"
...The robot’s behaviour was partially autonomous and partially remote controlled (Wizard-of-Oz, WoZ technique; see Gould et al. 1983; Dahlback et al. 1993; Maulsby et al. 1993), whereby the illusion is given to the subjects during the experiment that the robot operates fully autonomously....
[...]
Book•
28 Apr 2009TL;DR: In this paper, the authors provide an overview and instruction regarding the evaluation of interactive information retrieval systems with users and present core instruments and data collection techniques and measures, as well as a discussion of outstanding challenges and future research directions.
Abstract: This paper provides overview and instruction regarding the evaluation of interactive information retrieval systems with users The primary goal of this article is to catalog and compile material related to this topic into a single source This article (1) provides historical background on the development of user-centered approaches to the evaluation of interactive information retrieval systems; (2) describes the major components of interactive information retrieval system evaluation; (3) describes different experimental designs and sampling strategies; (4) presents core instruments and data collection techniques and measures; (5) explains basic data analysis techniques; and (4) reviews and discusses previous studies This article also discusses validity and reliability issues with respect to both measures and methods, presents background information on research ethics and discusses some ethical issues which are specific to studies of interactive information retrieval (IIR) Finally, this article concludes with a discussion of outstanding challenges and future research directions
565 citations
Book•
01 Jan 2002TL;DR: This chapter will review the main types of multimodal interfaces, their advantages and cognitive science underpinnings, primary features and architectural characteristics, and general research in the field of multi-modal interaction and interface design.
Abstract: Multimodal systems process two or more combined user input modes— such as speech, pen, touch, manual gestures, gaze, and head and body movements— in a coordinated manner with multimedia system output. This class of systems represents a new direction for computing, and a paradigm shift away from conventional WIMP interfaces. Since the appearance of Bolt’s (1980) “Put That There” demonstration system, which processed speech in parallel with touch-pad pointing, a variety of new multimodal systems has emerged. This new class of interfaces aims to recognize naturally occurring forms of human language and behavior, which incorporate at least one recognitionbased technology (e.g., speech, pen, vision). The development of novel multimodal systems has been enabled by the myriad input and output technologies currently becoming available, including new devices and improvements in recognition-based technologies. This chapter will review the main types of multimodal interfaces, their advantages and cognitive science underpinnings, primary features and architectural characteristics, and general research in the field of multimodal interaction and interface design.
528 citations
28 Jul 2012
TL;DR: This work systematically reviews how researchers conducted Wizard of Oz experiments published in the primary HRI publication venues from 2001 -- 2011 and proposes new reporting guidelines to aid future research.
Abstract: Many researchers use Wizard of Oz (WoZ) as an experimental technique, but there are methodological concerns over its use, and no comprehensive criteria on how to best employ it. We systematically review 54 WoZ experiments published in the primary HRI publication venues from 2001 -- 2011. Using criteria proposed by Fraser and Gilbert (1991), Green et al. (2004), Steinfeld et al. (2009), and Kelley (1984), we analyzed how researchers conducted HRI WoZ experiments. Researchers mainly used WoZ for verbal (72.2%) and non-verbal (48.1%) processing. Most constrained wizard production (90.7%), but few constrained wizard recognition (11%). Few reported measuring wizard error (3.7%), and few reported pre-experiment wizard training (5.4%). Few reported using WoZ in an iterative manner (24.1%). Based on these results we propose new reporting guidelines to aid future research.
456 citations
Cites background from "Wizard of Oz studies: why and how"
...These questions were motivated by Green et al. (2004), Dahlbäck et al. (1993), Fraser and Gilbert (1991), and Lohse (2010)....
[...]
...These questions were motivated by Kelley (1984), Fraser and Gilbert (1991), Green et al. (2004), Dahlbäck et al. (1993), and Steinfeld et al. (2009)....
[...]
...Dahlbäck et al. (1993) and Fraser and Gilbert (1991) stress the importance of scenario constraints for NLP WoZ experiments, as do Green, Hüttenrauch, and Eklundh (2004) for HRI....
[...]
01 Mar 2007
441 citations
References
More filters
TL;DR: This paper found that the 4-year-old adjusted his speech with regard to the changing capacities of different-aged listeners, and the younger the 2-year old, the greater was the observed speech adjustment.
Abstract: SHATZ, MARILYN, and GELMAN, ROCHEL. The Development of Communication Skills: Modifications in the Speech of Young Children as a Function of Listener. Monographs of the Society for Research in Child Development, 1973, 38 (5, Serial No. 152). This Monograph reports three studies of the 4-year-old's ability to adjust to a listener. In the first study (Study A) 16 Ss were pretested on modified versions of standard tests of "egocentrism." Following these, the children were asked first to tell an adult about a toy and then tell a 2-year-old about that toy. The eight Ss who had 2-year-old siblings were run on the toy task twice: once in an adult-sibling session, and once in an adult-nonsibling session. Finally, tapes were made of spontaneous conversations between the Ss and their mothers. As expected, the Ss performed poorly on the tests of "egocentrism." In contrast, Ss adjusted their speech production to their different listeners. Speech to 2-year-olds contained more short, simple utterances and more attentional utterances. The younger the 2-year-old, the greater was the observed speech adjustment. All children adjusted their speech whether or not they had younger siblings. In Study B tapes of uncontrolled conversations of five 4-year-olds each talking to a 2-year-old and an adult were obtained. Analyses of speech adjustments revealed a pattern of results like those of Study A. In Study C tapes of eight 4-year-olds talking to peers were collected and compared with the taped conversations with their mothers obtained in the first study. Analyses of the peer-directed versus adult-directed speech showed that, with respect to utterance length, the use of various constructions, and attentional utterances, peers were treated like adults. These results, combined with those of Study A, indicate that the 4-year-old adjusts his speech with regard to the changing capacities of different-aged listeners. The results of these studies are discussed with regard to previous work on the preschooler's communication skills and the variety of listener-produced cues that may influence the 4-year-old's tendency to "talk down." The implications of speaker-listener interaction for the process of language acquisition are considered. This content downloaded from 157.55.39.163 on Wed, 21 Sep 2016 04:38:28 UTC All use subject to http://about.jstor.org/terms
834 citations
TL;DR: The results throw doubt on the suggestion that the Late Closure strategy is favoured (in English) mainly because it is efficient in information processing terms and suggest that different languages make use of parsing strategies in an essentially arbitrary way.
441 citations
TL;DR: The “Wizard of Oz” technique for simulating future interactive technology and a partial taxonomy of such simulations is reviewed and a general Wizard of Oz methodology is suggested.
425 citations
"Wizard of Oz studies: why and how" refers background in this paper
...Fraser and Gilbert [ 9 ] in their simulations of speech-systems also propose the use of scenarios to achieve nxdistic interactions....
[...]