scispace - formally typeset
Search or ask a question
Proceedings ArticleDOI

Experiences of an In-Service Wizard-of-Oz Data Collection for the Deployment of a Call-Routing Application

Mats Wirén1, Robert Eklund1
26 Apr 2007-pp 56-63
TL;DR: This paper describes the experiences of collecting a corpus of 42,000 dialogues for a call-routing application using a Wizard-of-Oz approach, and provides a detailed exposition of the data collection as such and the application used, and compares the approach to methods previously used.
Abstract: This paper describes our experiences of collecting a corpus of 42,000 dialogues for a call-routing application using a Wizard-of-Oz approach. Contrary to common practice in the industry, we did not use the kind of automated application that elicits some speech from the customers and then sends all of them to the same destination, such as the existing touch-tone menu, without paying attention to what they have said. Contrary to the traditional Wizard-of-Oz paradigm, our data-collection application was fully integrated within an existing service, replacing the existing touch-tone navigation system with a simulated call-routing system. Thus, the subjects were real customers calling about real tasks, and the wizards were service agents from our customer care. We provide a detailed exposition of the data collection as such and the application used, and compare our approach to methods previously used.

Content maybe subject to copyright    Report

Citations
More filters
Proceedings ArticleDOI
01 Apr 2014
TL;DR: A framework for multi-lingual dialog research is presented, which combines speech recognition and synthesis with WOZ, and all components are open source and adaptable to different application scenarios.
Abstract: Wizard of Oz (WOZ) prototyping employs a human wizard to simulate anticipated functions of a future system. In Natural Language Processing this method is usually used to obtain early feedback on dialogue designs, to collect language corpora, or to explore interaction strategies. Yet, existing tools often require complex client-server configurations and setup routines, or suffer from compatibility problems with different platforms. Integrated solutions, which may also be used by designers and researchers without technical background, are missing. In this paper we present a framework for multi-lingual dialog research, which combines speech recognition and synthesis with WOZ. All components are open source and adaptable to different application scenarios.

7 citations


Cites background from "Experiences of an In-Service Wizard..."

  • ...It furthermore supports the collection of domain specific language corpora and the easy exploration of varying dialog designs (Wirén et al., 2007)....

    [...]

  • ...It furthermore supports the collection of domain specific language corpora and the easy exploration of varying dialog designs (Wirén et al., 2007)....

    [...]

26 Apr 2007
TL;DR: The purpose of this one-day workshop is to provide a forum to bring industrial and academic researchers together to share their experiences and visions in the dialog technology development, and to identify topics that are of interest to both camps.
Abstract: In the recent years, we have seen rapid adoption of dialog systems in commercial applications. They range from telephone-based services, in-car interactive systems, to online conversational service agents and talking characters in computer games. Openstandard platforms such as VoiceXML have been adopted by the industry, and become the driving force for the faster adoption of dialog applications. The widespread dialog applications in industry setting pose challenge for researchers in both industrial and academic worlds. Progress from academic world has not benefited the real world applications to a satisfactory extent. The purpose of this one-day workshop is to provide a forum to bring industrial and academic researchers together to share their experiences and visions in the dialog technology development, and to identify topics that are of interest to both camps. There are total 13 papers accepted for presentation at this workshop, with 8 papers for long presentation and 5 for short presentation. These papers are almost evenly divided between the industry and academic communities. In addition, two panels on the related dialog topics have been arranged during the workshop, with distinguished panelists of various backgrounds from academic, industrial, and standardization communities.

7 citations

Book ChapterDOI
01 Jan 2018
TL;DR: The discussion concludes that GUI-ii facilitates participation by relaxing demands for physical presence and by allowing people to participate from their own work environment while still making it easy for them to directly influence contents, structure and interaction.
Abstract: This chapter presents GUI-ii, Graphical User Interface interaction interview, a method used to remotely discuss, develop and test GUI prototypes with users and stakeholders. Examples of such sessions are presented to demonstrate that the main benefit of GUI-ii is that this way of co-designing allows for interaction-informed discussions around functions and user interfaces, where re-design and hands-on experience can be integrated and efficiently carried out remotely. Using a facilitation tool to enact GUI layout and responses allows participation and evaluation to be mixed in participatory design sessions in a productive way. This form of participatory design is discussed along the dimensions found in Sanders’ Map of Design Research. The discussion concludes that GUI-ii facilitates participation by relaxing demands for physical presence and by allowing people to participate from their own work environment while still making it easy for them to directly influence contents, structure and interaction.

6 citations

02 Apr 2014
TL;DR: In this article, a groupe de travail academique pour la scolarisation des eleves intellectuellement precoces publie une plaquette d'information a destination des etablissements scolaires.
Abstract: Le groupe de travail academique pour la scolarisation des eleves intellectuellement precoces publie une plaquette d'information a destination des etablissements scolaires. L'academie a mis en lien tout un ensemble d'informations : le formulaire de contact du referent eip academique, les colloques et autres rencontres de 2013 https://www.ac-aix-marseille.fr/ped... BESOINS EDUCATIFS PARTICULIERS 2013.indd - Onisep (...)

5 citations

DOI
01 Jan 2009
TL;DR: It is concluded that natural language interfaces to meeting archives are useful, but that more experimental work is needed to find ways to incent users to take advantage of the expressive power of natural language when asking questions about meetings.
Abstract: Retrieving information from archived meetings is a new domain of information retrieval that has received increasing attention in the past few years. Search in spontaneous spoken conversations has been recognized as more difficult than text-based document retrieval because meeting discussions contain two levels of information: the content itself, i.e. what topics are discussed, but also the argumentation process, i.e. what conflicts are resolved and what decisions are made. To capture the richness of information in meetings, current research focuses on recording meetings in Smart-Rooms, transcribing meeting discussion into text and annotating discussion with semantic higher-level structures to allow for efficient access to the data. However, it is not yet clear what type of user interface is best suited for searching and browsing such archived, annotated meetings. Content-based retrieval with keyword search is too naive and does not take into account the semantic annotations on the data. The objective of this thesis is to assess the feasibility and usefulness of a natural language interface to meeting archives that allows users to ask complex questions about meetings and retrieve episodes of meeting discussions based on semantic annotations. The particular issues that we address are: the need of argumentative annotation to answer questions about meetings; the linguistic and domain-specific natural language understanding techniques required to interpret such questions; and the use of visual overviews of meeting annotations to guide users in formulating questions. To meet the outlined objectives, we have annotated meetings with argumentative structure and built a prototype of a natural language understanding engine that interprets questions based on those annotations. Further, we have performed two sets of user experiments to study what questions users ask when faced with a natural language interface to annotated meeting archives. For this, we used a simulation method called Wizard of Oz, to enable users to express questions in their own terms without being influenced by limitations in speech recognition technology. Our experimental results show that technically it is feasible to annotate meetings and implement a deep-linguistic NLU engine for questions about meetings, but in practice users do not consistently take advantage of these features. Instead they often search for keywords in meetings. When visual overviews of the available annotations are provided, users refer to those annotations in their questions, but the complexity of questions remains simple. Users search with a breadth-first approach, asking questions in sequence instead of a single complex question. We conclude that natural language interfaces to meeting archives are useful, but that more experimental work is needed to find ways to incent users to take advantage of the expressive power of natural language when asking questions about meetings.

4 citations

References
More filters
Proceedings ArticleDOI
01 Feb 1993
TL;DR: It is concluded that empirical studies of the unique qualities of man-machine interaction as distinct from general human discourse are required for the development of user-friendly interactive systems.
Abstract: Current approaches to the development of natural language dialogue systems are discussed, and it is claimed that they do not sufficiently consider the unique qualities of man-machine interaction as distinct from general human discourse. It is concluded that empirical studies of this unique communication situation are required for the development of user-friendly interactive systems. One way of achieving this is through the use of so-called Wizard of Oz studies. The focus of the work described in the paper is on the practical execution of the studies and the methodological conclusions drawn on the basis of the authors' experience. While the focus is on natural language interfaces, the methods used and the conclusions drawn from the results obtained are of relevance also to other kinds of intelligent interfaces.

892 citations

Journal ArticleDOI
TL;DR: This paper focuses on the task of automatically routing telephone calls based on a user's fluently spoken response to the open-ended prompt of “ How may I help you? ”.

664 citations


"Experiences of an In-Service Wizard..." refers background or methods in this paper

  • ...One exception from this is the data collection for the original AT&T “How May I Help You” system (Gorin et al. 1997; Ammicht et al. 1999), which comprised three batches of transactions with live customers, each involving up to 12,000 utterances....

    [...]

  • ...The sole such data collection that we are aware of was made for the original AT&T “How May I Help you” system (Gorin et al. 1997; Ammicht et al. 1999)....

    [...]

Journal ArticleDOI
TL;DR: The focus of the work described in the paper is on the practical execution of the studies and the methodological conclusions drawn on the basis of the authors' experience, and the methods used and the conclusions drawn are of relevance also to other kinds of intelligent interfaces.
Abstract: Current approaches to the development of natural language dialogue systems are discussed, and it is claimed that they do not sufficiently consider the unique qualities of man-machine interaction as...

641 citations

Journal ArticleDOI
TL;DR: The “Wizard of Oz” technique for simulating future interactive technology and a partial taxonomy of such simulations is reviewed and a general Wizard of Oz methodology is suggested.

425 citations

Proceedings ArticleDOI
21 Mar 1993
TL;DR: This work focuses here on selection of training and test data, evaluation of language understanding, and the continuing search for evaluation methods that will correlate well with expected performance of the technology in applications.
Abstract: The Air Travel Information System (ATIS) domain serves as the common task for DARPA spoken language system research and development The approaches and results possible in this rapidly growing area are structured by available corpora, annotations of that data, and evaluation methods Coordination of this crucial infrastructure is the charter of the Multi-Site ATIS Data COllection Working group (MADCOW) We focus here on selection of training and test data, evaluation of language understanding, and the continuing search for evaluation methods that will correlate well with expected performance of the technology in applications

206 citations


"Experiences of an In-Service Wizard..." refers background in this paper

  • ...Other well-known instances are “Voyager” (Zue et al. 1989) and the individual ATIS collections (Hirschman et al. 1993) which involved up to a hundred subjects or (again) up to 12,000 utterances....

    [...]

  • ...1989) and the individual ATIS collections (Hirschman et al. 1993) which involved up to a hundred subjects or (again) up to 12,000 utterances....

    [...]