scispace - formally typeset
Search or ask a question
Proceedings ArticleDOI

User Experience with Smart Voice Assistants: The Accent Perspective

TL;DR: Results suggest no significant differences in usability between the two user groups, however the same being significant for the satisfaction levels.
Abstract: Voice assistants (VA) like Siri, Alexa, Google Assistant, Cortana, and Bixby are increasingly becoming popular among the general mass. They are being used by a large group of people who use English as their primary mode of communication (native speakers) as well as those who are secondary English language speakers (non-native speakers). However, from an end-user perspective very little is known about the usability, acceptability, satisfaction, and the usage pattern of the VA's between these two different group of users. The current study aims to identify if there exist any differences between these two group of users with respect to the overall usability and the satisfaction received after using the VA's. To answer the research questions a mixed methodology approach is undertaken comprising of an online questionnaire survey in conjunction with a real-world testing of the VA's. The experiment is conducted in Thailand with 275 users for the questionnaire phase and 52 users for the testing phase from 7 different countries. Siri and Alexa are used as the VA representatives since they are the two most popular ones in Thailand. Results suggest no significant differences in usability between the two user groups, however the same being significant for the satisfaction levels.
Citations
More filters
Journal ArticleDOI
TL;DR: An in-depth comparative analysis is conducted for determining which model best explains the user's acceptance of using VAs, finding that enjoyment has the greatest influence and the effect of usefulness is found to be less, which suggests the existence of some usability issues with the VAs.
Abstract: Rapid enhancements in the Internet of Things (IoT) and other technologies have resulted in the emergence of various types of smart IT products such as the voice assistants (VAs). Accordingly, several attempts have been made through extant research for explaining the acceptance of these devices by using different models related to technology acceptance. In this article, an in-depth comparative analysis is conducted for determining which model best explains the user’s acceptance of using VAs. The technology acceptance model (TAM), theory of planned behavior (TPB), unified theory of acceptance and use of technology (UTAUT), and value-based adoption model (VAM) are used for the purpose of comparison using data collected from 436 (275 potential and 161 actual) participants from the Amazon Mechanical Turk (MTurk) platform. A maximum-likelihood structural equation modelling approach is used for the purpose of hypothesis testing on a model-by-model basis, while for the purpose of comparison between the models, Hotelling’s ${ T}^{ 2}$ test is used as the statistical measure. All the hypotheses across all the models are found to be true, though with varying degrees of ${\mathrm{ adjusted-}}{ R}^{ 2}$ values. VAM is found to have the greatest predictive power ( ${\mathrm{ adjusted-}}{ R}^{ 2} ={ 0}.{ 6}{ 83}$ ) and TAM has the least predictive power ( ${\mathrm{ adjusted-}}{ R}^{ 2} ={ 0}.{ 439}$ ) for behavioral intention (BI). Additionally, a multiple regression analysis is conducted for comparing each of the factors considered in the models in terms of their influence on BI. Results show that enjoyment has the greatest influence (31.24%), followed by subjective norms (15.43%). The effect of usefulness is found to be less (11.86%), which suggests the existence of some usability issues with the VAs. Finally, the research implications are discussed and suggestions are provided.

39 citations

Journal ArticleDOI
01 Jan 2021
TL;DR: In this paper, a subjective scale called the Voice Usability Scale (VUS) was proposed to evaluate the suitability of SUS for usability evaluation of voice-assistants.
Abstract: Currently, the use of voice-assistants has been on the rise, but a user-centric usability evaluation of these devices is a must for ensuring their success. System Usability Scale (SUS) is one such popular usability instrument in a Graphical User Interface (GUI) scenario. However, there are certain fundamental differences between GUI and voice-based systems, which makes it uncertain regarding the suitability of SUS in a voice scenario. The present work has a twofold objective: to check the suitability of SUS for usability evaluation of voice-assistants and developing a subjective scale in line with SUS that considers the unique aspects of voice-based communication. We call this scale as the Voice Usability Scale (VUS). For fulfilling the objectives, a subjective test is conducted with 62 participants. An Exploratory Factor Analysis suggests that SUS has a number of drawbacks for measuring the voice usability. Moreover, in case of VUS, the most optimal factor structure identifies three main components: usability, affective, and recognizability and visibility. The current findings should provide an initial starting point to form a useful theoretical and practical basis for subjective usability assessment of voice-based systems.

22 citations

Journal ArticleDOI
TL;DR: The results show that usefulness, ease of use, compatibility and perceived complementarity have significant positive effects on the purchase intention of the VA market and the senior consumers are more likely to purchase the VAs and other allied smart-home devices within a given time frame when compared to the younger consumers.
Abstract: The market for voice assistants (VAs) and other allied voice-based smart-home products is gradually emerging. The initial growth has been slower than expected; therefore, an in-depth simultaneous intention and diffusion analysis is needed for identifying the relevant factors along with finding out the target consumers. This work uses technology acceptance model as the core for analyzing the adoption intention of the VA-based system and extends it with three additional factors: compatibility, perceived complementarity and privacy concerns. The diffusion analysis for the same is done using the multivariate probit model. Certain characteristics of the VA-based systems like the network effects between the products/services and the importance of protecting personal information are considered in this work, apart from various demographic variables like age, gender, income and education levels. Two separate surveys were conducted for the purpose of data collection from 315 and 1945 participants residing in Thailand for analyzing the adoption and diffusion scenario, respectively. The results show that usefulness, ease of use, compatibility and perceived complementarity have significant positive effects on the purchase intention. In terms of diffusion of the VA market, unlike other Information Communication Technology-based products/services, the results show that the senior consumers are more likely to purchase the VAs and other allied smart-home devices within a given time frame when compared to the younger consumers. Therefore, new strategies should be developed that promote the usage of VAs by the young population for increasing the market demand.

22 citations


Cites background from "User Experience with Smart Voice As..."

  • ...Similarly, the authors in [22] test some of the commercially available VAs to see how well they can understand the English commands from the native and nonnative English language speakers....

    [...]

Proceedings ArticleDOI
08 May 2021
TL;DR: In this paper, the authors surveyed the ACM and IEEE literatures to determine which quantitative measures and measurements have been deemed important for voice UX and found that there is little consensus, even with similar situations and systems, as well as an overreliance on lab work and unvalidated scales.
Abstract: Computer voice is experiencing a renaissance through the growing popularity of voice-based interfaces, agents, and environments. Yet, how to measure the user experience (UX) of voice-based systems remains an open and urgent question, especially given that their form factors and interaction styles tend to be non-visual, intangible, and often considered disembodied or “body-less.” As a first step, we surveyed the ACM and IEEE literatures to determine which quantitative measures and measurements have been deemed important for voice UX. Our findings show that there is little consensus, even with similar situations and systems, as well as an overreliance on lab work and unvalidated scales. In response, we offer two high-level descriptive frameworks for guiding future research, developing standardized instruments, and informing ongoing review work. Our work highlights the current strengths and weaknesses of voice UX research and charts a path towards measuring voice UX in a more comprehensive way.

15 citations

Proceedings ArticleDOI
TL;DR: In this article, the authors surveyed the ACM and IEEE literatures to determine which quantitative measures and measurements have been deemed important for voice UX and found that there is little consensus, even with similar situations and systems, as well as an overreliance on lab work and unvalidated scales.
Abstract: Computer voice is experiencing a renaissance through the growing popularity of voice-based interfaces, agents, and environments. Yet, how to measure the user experience (UX) of voice-based systems remains an open and urgent question, especially given that their form factors and interaction styles tend to be non-visual, intangible, and often considered disembodied or "body-less." As a first step, we surveyed the ACM and IEEE literatures to determine which quantitative measures and measurements have been deemed important for voice UX. Our findings show that there is little consensus, even with similar situations and systems, as well as an overreliance on lab work and unvalidated scales. In response, we offer two high-level descriptive frameworks for guiding future research, developing standardized instruments, and informing ongoing review work. Our work highlights the current strengths and weaknesses of voice UX research and charts a path towards measuring voice UX in a more comprehensive way.

13 citations

References
More filters
Journal ArticleDOI
TL;DR: Results from the analysis of this large number of SUS scores show that the SUS is a highly robust and versatile tool for usability professionals.
Abstract: This article presents nearly 10 year's worth of System Usability Scale (SUS) data collected on numerous products in all phases of the development lifecycle. The SUS, developed by Brooke (1996), reflected a strong need in the usability community for a tool that could quickly and easily collect a user's subjective rating of a product's usability. The data in this study indicate that the SUS fulfills that need. Results from the analysis of this large number of SUS scores show that the SUS is a highly robust and versatile tool for usability professionals. The article presents these results and discusses their implications, describes nontraditional uses of the SUS, explains a proposed modification to the SUS to provide an adjective rating that correlates with a given score, and provides details of what constitutes an acceptable SUS score.

3,192 citations


"User Experience with Smart Voice As..." refers result in this paper

  • ...This is in accordance to the findings from previous research in [29]....

    [...]

Journal Article
TL;DR: A seven-point adjective-anchored Likert scale was added as an eleventh question to nearly 1,000 System Usability Scale (SUS) surveys as mentioned in this paper.
Abstract: The System Usability Scale (SUS) is an inexpensive, yet effective tool for assessing the usability of a product, including Web sites, cell phones, interactive voice response systems, TV applications, and more. It provides an easy-to-understand score from 0 (negative) to 100 (positive). While a 100-point scale is intuitive in many respects and allows for relative judgments, information describing how the numeric score translates into an absolute judgment of usability is not known. To help answer that question, a seven-point adjective-anchored Likert scale was added as an eleventh question to nearly 1,000 SUS surveys. Results show that the Likert scale scores correlate extremely well with the SUS scores (r=0.822). The addition of the adjective rating scale to the SUS may help practitioners interpret individual SUS scores and aid in explaining the results to non-human factors professionals.

2,592 citations


"User Experience with Smart Voice As..." refers methods in this paper

  • ...The SUS scores which are obtained on a 5-point scale (5 = “Strongly agree” and 1 = “Strongly disagree”) are converted to an adjective rating scale that has been used in previous studies to map the semantic meaning to the SUS scores [28]....

    [...]

Journal Article
TL;DR: A questionnaire that could be used to take a quick measurement of how people perceived the usability of computer systems on which they were working, SUS proved to be an extremely simple and reliable tool for use when doing usability evaluations.
Abstract: Rather more than 25 years ago, as part of a usability engineering program, I developed a questionnaire---the System Usability Scale (SUS)---that could be used to take a quick measurement of how people perceived the usability of computer systems on which they were working. This proved to be an extremely simple and reliable tool for use when doing usability evaluations, and I decided, with the blessing of engineering management at Digital Equipment Co. Ltd (DEC; where I developed SUS), that it was probably something that could be used by other organizations (the benefit for us being that if they did use it, we potentially had something we could use to compare their systems against ours). So, in 1986, I made SUS freely available to a number of colleagues, with permission to pass it on to anybody else who might find it useful, and over the next few years occasionally heard of evaluations of systems where researchers and usability engineers had used it with some success.

1,265 citations


"User Experience with Smart Voice As..." refers methods in this paper

  • ...The ten pointers of the SUS scale is presented in Table I....

    [...]

  • ...LIST OF 10 MOST FREQUENT TASKS Task Number Task Description 1 Use the VA for checking the current weather 2 Ask the VA to get directions to commute between two places 3 Use the VA to check daily news headlines 4 Use the VA to play songs 5 Use the VA for setting up an alarm for a specific time 6 Ask the VA about a famous personality 7 Ask the VA to adjust its volume 8 Ask the VA to tell some jokes 9 Using the VA to read audiobook 10 Use the VA to read and compose e-mails 10th ICCCNT 2019 July 6-8, 2019, IIT - Kanpur, Kanpur, India The SUS scores and the adjective rating scale have a high correlation (Pearson’s Correlation Coefficient = 0.97) as evident from Fig....

    [...]

  • ...After completing each task, the participants are asked to fill up a digital form, which contains the ten items of the SUS scale....

    [...]

  • ...4 and 5, we show the average SUS scores for the two groups of speakers across each of the ten SUS items for Siri and Alexa respectively....

    [...]

  • ...The SUS scores which are obtained on a 5-point scale (5 = “Strongly agree” and 1 = “Strongly disagree”) are converted to an adjective rating scale that has been used in previous studies to map the semantic meaning to the SUS scores [28]....

    [...]

Proceedings ArticleDOI
Ewa Luger1, Abigail Sellen1
07 May 2016
TL;DR: This paper reports the findings of interviews with 14 users of CAs in an effort to understand the current interactional factors affecting everyday use, and finds user expectations dramatically out of step with the operation of the systems.
Abstract: The past four years have seen the rise of conversational agents (CAs) in everyday life. Apple, Microsoft, Amazon, Google and Facebook have all embedded proprietary CAs within their software and, increasingly, conversation is becoming a key mode of human-computer interaction. Whilst we have long been familiar with the notion of computers that speak, the investigative concern within HCI has been upon multimodality rather than dialogue alone, and there is no sense of how such interfaces are used in everyday life. This paper reports the findings of interviews with 14 users of CAs in an effort to understand the current interactional factors affecting everyday use. We find user expectations dramatically out of step with the operation of the systems, particularly in terms of known machine intelligence, system capability and goals. Using Norman's 'gulfs of execution and evaluation' [30] we consider the implications of these findings for the design of future systems.

723 citations

Journal ArticleDOI
TL;DR: This paper deals with the creation of an 8 item short version of the UEQ, which is optimized for these specific application scenarios and first validations of this short version are described.
Abstract: The user experience questionnaire (UEQ) is a widely used questionnaire to measure the subjective impression of users towards the user experience of products. The UEQ is a semantic differential with 26 items. Filling out the UEQ takes approximately 3-5 minutes, i.e. the UEQ is already reasonably efficient concerning the time required to answer all items. However, there exist several valid application scenarios, where filling out the entire UEQ appears impractical. This paper deals with the creation of an 8 item short version of the UEQ, which is optimized for these specific application scenarios. First validations of this short version are also described.

385 citations


"User Experience with Smart Voice As..." refers methods in this paper

  • ...The ten pointers of the SUS scale is presented in Table I....

    [...]

  • ...LIST OF 10 MOST FREQUENT TASKS Task Number Task Description 1 Use the VA for checking the current weather 2 Ask the VA to get directions to commute between two places 3 Use the VA to check daily news headlines 4 Use the VA to play songs 5 Use the VA for setting up an alarm for a specific time 6 Ask the VA about a famous personality 7 Ask the VA to adjust its volume 8 Ask the VA to tell some jokes 9 Using the VA to read audiobook 10 Use the VA to read and compose e-mails 10th ICCCNT 2019 July 6-8, 2019, IIT - Kanpur, Kanpur, India The SUS scores and the adjective rating scale have a high correlation (Pearson’s Correlation Coefficient = 0.97) as evident from Fig....

    [...]

  • ...After completing each task, the participants are asked to fill up a digital form, which contains the ten items of the SUS scale....

    [...]

  • ...4 and 5, we show the average SUS scores for the two groups of speakers across each of the ten SUS items for Siri and Alexa respectively....

    [...]

  • ...The SUS scores which are obtained on a 5-point scale (5 = “Strongly agree” and 1 = “Strongly disagree”) are converted to an adjective rating scale that has been used in previous studies to map the semantic meaning to the SUS scores [28]....

    [...]

Trending Questions (1)
How smart experiences build service loyalty: The importance of consumer love for smart voice assistants.?

The provided information does not address the topic of how smart experiences build service loyalty or the importance of consumer love for smart voice assistants.