Experiences of an In-Service Wizard-of-Oz Data Collection for the Deployment of a Call-Routing Application
26 Apr 2007-pp 56-63
TL;DR: This paper describes the experiences of collecting a corpus of 42,000 dialogues for a call-routing application using a Wizard-of-Oz approach, and provides a detailed exposition of the data collection as such and the application used, and compares the approach to methods previously used.
Abstract: This paper describes our experiences of collecting a corpus of 42,000 dialogues for a call-routing application using a Wizard-of-Oz approach. Contrary to common practice in the industry, we did not use the kind of automated application that elicits some speech from the customers and then sends all of them to the same destination, such as the existing touch-tone menu, without paying attention to what they have said. Contrary to the traditional Wizard-of-Oz paradigm, our data-collection application was fully integrated within an existing service, replacing the existing touch-tone navigation system with a simulated call-routing system. Thus, the subjects were real customers calling about real tasks, and the wizards were service agents from our customer care. We provide a detailed exposition of the data collection as such and the application used, and compare our approach to methods previously used.
Citations
More filters
01 Jan 2010
TL;DR: The frequency and distribution of filledpauses (FPs) in ecologically valid data where unaware and Authentic customers called in to report problems with theirphony and/or Internet services and were met by a novel Wizard-of-Oz paradigm using real call center agents as Wizards of Oz wizards are studied.
Abstract: This paper studies the frequency and distribution of filledpauses (FPs) in ecologically valid data where unaware andauthentic customers called in to report problems with theirtelephony and/or Internet services and were met by a novelWizard-of-Oz paradigm using real call center agents aswizards. The data analyzed were caller utterances followinga directed or an open disambiguation prompt. While nosignificant differences in FP production were observed as afunction of prompt type, FP frequency was found to beconsiderably higher than what is usually reported in theliterature. Moreover, a higher proportion of utterance-initialFPs than normally reported was also observed. The results arecompared to previously reported FP frequencies. Potentialimplications for data collection methodology are discussed.
4 citations
Additional excerpts
...The data collection is described in detail in [22]....
[...]
Dissertation•
17 Sep 2012TL;DR: Despite the difficulty of synchronising evaluators’ individual expertise and experiences was admitted, the research findings suggested practical measurements to address the individual differences at acceptable levels through applying additional assistance and constraints to evaluator's system facilitation judgement and execution.
Abstract: Wizard-of-Oz (WoZ) is a flexible, efficient and cost-economic method to the design and evaluation of interaction systems, particularly such of natural dialogue and smart systems. However, the literature review in the beginning of this research indicated that researchers struggled to implement WoZ and be able to gain reliable and valid experimental results in terms of system facilitation consistency; and WoZ has been criticised for a lack of systematic assessment of influence variables, especially when it was used to study new emerging information and communication technologies. Hence, this research aimed to understand and improve the reliability and validity of WoZ.
The research consisted of a series of empirical studies to incrementally deepen the understanding of influence variables. The main body of research comprised studies investigating (1) the impact of schema as WoZ study guidelines, (2) the impact of control panel in system facilitation, (3) the variables affecting evaluator’s interpretation of schema, control panel and subject activity, and (4) the differences in multiple evaluators’ system facilitation.
The results indicated that neither rigorous nor general schemas supported highly reliable system facilitation; rather, schemas should be accordingly proposed on the base of predictable or unpredictable user interactions. Also the results revealed the hidden relationships between control panel and system facilitation through identifying the control panel influence factors such like layouts and functions and their connections with system facilitation. Additionally, despite the difficulty of synchronising evaluators’ individual expertise and experiences was admitted, the research findings suggested practical measurements to address the individual differences at acceptable levels through applying additional assistance and constraints to evaluator’s system facilitation judgement and execution. And the results also provided secondary understanding towards smart system design for domestic communication and the development of WoZ system.
4 citations
Dissertation•
01 Jan 2013
3 citations
TL;DR: In this article , a method for the co-design of mobile navigation applications in the context of use was explored, in which the user is actively involved as a codesigner, which enables the continuous evaluation of design suggestions.
Abstract: This study explores a method for the co-design of mobile applications in the context of use. In 36 sessions with future users, synchronous co-design of a mobile navigation application was conducted in the intended use environment – a hospital – using an interactive Wizard-of-Oz-controlled prototype. The results show that co-design in the intended use environment contributes to the elicitation of design suggestions. Concerning the co-design method, the results show that by using interactive prototyping the user is actively involved as a co-designer, which empowers the user and enables the continuous evaluation of design suggestions.
1 citations
01 Jan 2015
TL;DR: Analysis of the tree-structures of each respective method for the retrieved input shows that, while both methods are implementable, constituency parsing is to be preferred due to the stable structuring of the words in the received commands.
Abstract: The purpose of this paper is to compare two general methods of parsing; dependency parsing and constituency parsing. This is to examine which is favorable when interpreting input in the form of natural language regarding navigating in a simulated environment. Accordingly, we collected a series of different input commands using a form of Wizard-Of-Oz-system were the participants gave commands to what they believed was a fully automated application. Analysis of the tree-structures of each respective method for the retrieved input shows that, while both methods are implementable, constituency parsing is to be preferred due to the stable structuring of the words in the received commands.
Cites background from "Experiences of an In-Service Wizard..."
...Due to Wizard-of-Oz-related studies being time consuming [10] a limit on the test cases had to be set which might not prove to be optimal....
[...]
References
More filters
01 Feb 1993
TL;DR: It is concluded that empirical studies of the unique qualities of man-machine interaction as distinct from general human discourse are required for the development of user-friendly interactive systems.
Abstract: Current approaches to the development of natural language dialogue systems are discussed, and it is claimed that they do not sufficiently consider the unique qualities of man-machine interaction as distinct from general human discourse. It is concluded that empirical studies of this unique communication situation are required for the development of user-friendly interactive systems. One way of achieving this is through the use of so-called Wizard of Oz studies. The focus of the work described in the paper is on the practical execution of the studies and the methodological conclusions drawn on the basis of the authors' experience. While the focus is on natural language interfaces, the methods used and the conclusions drawn from the results obtained are of relevance also to other kinds of intelligent interfaces.
892 citations
[...]
TL;DR: This paper focuses on the task of automatically routing telephone calls based on a user's fluently spoken response to the open-ended prompt of “ How may I help you? ”.
664 citations
"Experiences of an In-Service Wizard..." refers background or methods in this paper
...One exception from this is the data collection for the original AT&T “How May I Help You” system (Gorin et al. 1997; Ammicht et al. 1999), which comprised three batches of transactions with live customers, each involving up to 12,000 utterances....
[...]
...The sole such data collection that we are aware of was made for the original AT&T “How May I Help you” system (Gorin et al. 1997; Ammicht et al. 1999)....
[...]
TL;DR: The focus of the work described in the paper is on the practical execution of the studies and the methodological conclusions drawn on the basis of the authors' experience, and the methods used and the conclusions drawn are of relevance also to other kinds of intelligent interfaces.
Abstract: Current approaches to the development of natural language dialogue systems are discussed, and it is claimed that they do not sufficiently consider the unique qualities of man-machine interaction as...
641 citations
TL;DR: The “Wizard of Oz” technique for simulating future interactive technology and a partial taxonomy of such simulations is reviewed and a general Wizard of Oz methodology is suggested.
425 citations
21 Mar 1993
TL;DR: This work focuses here on selection of training and test data, evaluation of language understanding, and the continuing search for evaluation methods that will correlate well with expected performance of the technology in applications.
Abstract: The Air Travel Information System (ATIS) domain serves as the common task for DARPA spoken language system research and development The approaches and results possible in this rapidly growing area are structured by available corpora, annotations of that data, and evaluation methods Coordination of this crucial infrastructure is the charter of the Multi-Site ATIS Data COllection Working group (MADCOW) We focus here on selection of training and test data, evaluation of language understanding, and the continuing search for evaluation methods that will correlate well with expected performance of the technology in applications
206 citations
"Experiences of an In-Service Wizard..." refers background in this paper
...Other well-known instances are “Voyager” (Zue et al. 1989) and the individual ATIS collections (Hirschman et al. 1993) which involved up to a hundred subjects or (again) up to 12,000 utterances....
[...]
...1989) and the individual ATIS collections (Hirschman et al. 1993) which involved up to a hundred subjects or (again) up to 12,000 utterances....
[...]