scispace - formally typeset
Search or ask a question
Journal ArticleDOI

An Empathy-Driven, Conversational Artificial Intelligence Agent (Wysa) for Digital Mental Well-Being: Real-World Data Evaluation Mixed-Methods Study

23 Nov 2018-Jmir mhealth and uhealth (JMIR Publications Inc.)-Vol. 6, Iss: 11
TL;DR: A preliminary real-world data evaluation of the effectiveness and engagement levels of an AI-enabled, empathetic, text-based conversational mobile mental well-being app, Wysa, on users with self-reported symptoms of depression shows promise.
Abstract: Background: A World Health Organization 2017 report stated that major depression affects almost 5% of the human population. Major depression is associated with impaired psychosocial functioning and reduced quality of life. Challenges such as shortage of mental health personnel, long waiting times, perceived stigma, and lower government spends pose barriers to the alleviation of mental health problems. Face-to-face psychotherapy alone provides only point-in-time support and cannot scale quickly enough to address this growing global public health challenge. Artificial intelligence (AI)-enabled, empathetic, and evidence-driven conversational mobile app technologies could play an active role in filling this gap by increasing adoption and enabling reach. Although such a technology can help manage these barriers, they should never replace time with a health care professional for more severe mental health problems. However, app technologies could act as a supplementary or intermediate support system. Mobile mental well-being apps need to uphold privacy and foster both short- and long-term positive outcomes. Objective: This study aimed to present a preliminary real-world data evaluation of the effectiveness and engagement levels of an AI-enabled, empathetic, text-based conversational mobile mental well-being app, Wysa, on users with self-reported symptoms of depression. Methods: In the study, a group of anonymous global users were observed who voluntarily installed the Wysa app, engaged in text-based messaging, and self-reported symptoms of depression using the Patient Health Questionnaire-9. On the basis of the extent of app usage on and between 2 consecutive screening time points, 2 distinct groups of users (high users and low users) emerged. The study used mixed-methods approach to evaluate the impact and engagement levels among these users. The quantitative analysis measured the app impact by comparing the average improvement in symptoms of depression between high and low users. The qualitative analysis measured the app engagement and experience by analyzing in-app user feedback and evaluated the performance of a machine learning classifier to detect user objections during conversations. Results: The average mood improvement (ie, difference in pre- and post-self-reported depression scores) between the groups (ie, high vs low users; n=108 and n=21, respectively) revealed that the high users group had significantly higher average improvement (mean 5.84 [SD 6.66]) compared with the low users group (mean 3.52 [SD 6.15]); Mann-Whitney P=.03 and with a moderate effect size of 0.63. Moreover, 67.7% of user-provided feedback responses found the app experience helpful and encouraging. Conclusions: The real-world data evaluation findings on the effectiveness and engagement levels of Wysa app on users with self-reported symptoms of depression show promise. However, further work is required to validate these initial findings in much larger samples and across longer periods.
Citations
More filters
Journal ArticleDOI
TL;DR: It is argued that embodied AI is a promising approach across the field of mental health; however, further research is needed to address the broader ethical and societal concerns of these technologies to negotiate best research and medical practices in innovative mental health care.
Abstract: Background: Research in embodied artificial intelligence (AI) has increasing clinical relevance for therapeutic applications in mental health services. With innovations ranging from ‘virtual psychotherapists’ to social robots in dementia care and autism disorder, to robots for sexual disorders, artificially intelligent virtual and robotic agents are increasingly taking on high-level therapeutic interventions that used to be offered exclusively by highly trained, skilled health professionals. In order to enable responsible clinical implementation, ethical and social implications of the increasing use of embodied AI in mental health need to be identified and addressed. Objective: This paper assesses the ethical and social implications of translating embodied AI applications into mental health care across the fields of Psychiatry, Psychology and Psychotherapy. Building on this analysis, it develops a set of preliminary recommendations on how to address ethical and social challenges in current and future applications of embodied AI. Methods: Based on a thematic literature search and established principles of medical ethics, an analysis of the ethical and social aspects of currently embodied AI applications was conducted across the fields of Psychiatry, Psychology, and Psychotherapy. To enable a comprehensive evaluation, the analysis was structured around the following three steps: assessment of potential benefits; analysis of overarching ethical issues and concerns; discussion of specific ethical and social issues of the interventions. Results: From an ethical perspective, important benefits of embodied AI applications in mental health include new modes of treatment, opportunities to engage hard-to-reach populations, better patient response, and freeing up time for physicians. Overarching ethical issues and concerns include: harm prevention and various questions of data ethics; a lack of guidance on development of AI applications, their clinical integration and training of health professionals; ‘gaps’ in ethical and regulatory frameworks; the potential for misuse including using the technologies to replace established services, thereby potentially exacerbating existing health inequalities. Specific challenges identified and discussed in the application of embodied AI include: matters of risk-assessment, referrals, and supervision; the need to respect and protect patient autonomy; the role of non-human therapy; transparency in the use of algorithms; and specific concerns regarding long-term effects of these applications on understandings of illness and the human condition. Conclusions: We argue that embodied AI is a promising approach across the field of mental health; however, further research is needed to address the broader ethical and societal concerns of these technologies to negotiate best research and medical practices in innovative mental health care. We conclude by indicating areas of future research and developing recommendations for high-priority areas in need of concrete ethical guidance.

212 citations

Journal ArticleDOI
TL;DR: There is an urgent need for a robust evaluation of diverse health care conversational agents’ formats, focusing on their acceptability, safety, and effectiveness.
Abstract: Background: Conversational agents, also known as chatbots, are computer programs designed to simulate human text or verbal conversations. They are increasingly used in a range of fields, including health care. By enabling better accessibility, personalization, and efficiency, conversational agents have the potential to improve patient care. Objective: This study aimed to review the current applications, gaps, and challenges in the literature on conversational agents in health care and provide recommendations for their future research, design, and application. Methods: We performed a scoping review. A broad literature search was performed in MEDLINE (Medical Literature Analysis and Retrieval System Online; Ovid), EMBASE (Excerpta Medica database; Ovid), PubMed, Scopus, and Cochrane Central with the search terms “conversational agents,” “conversational AI,” “chatbots,” and associated synonyms. We also searched the gray literature using sources such as the OCLC (Online Computer Library Center) WorldCat database and ResearchGate in April 2019. Reference lists of relevant articles were checked for further articles. Screening and data extraction were performed in parallel by 2 reviewers. The included evidence was analyzed narratively by employing the principles of thematic analysis. Results: The literature search yielded 47 study reports (45 articles and 2 ongoing clinical trials) that matched the inclusion criteria. The identified conversational agents were largely delivered via smartphone apps (n=23) and used free text only as the main input (n=19) and output (n=30) modality. Case studies describing chatbot development (n=18) were the most prevalent, and only 11 randomized controlled trials were identified. The 3 most commonly reported conversational agent applications in the literature were treatment and monitoring, health care service support, and patient education. Conclusions: The literature on conversational agents in health care is largely descriptive and aimed at treatment and monitoring and health service support. It mostly reports on text-based, artificial intelligence–driven, and smartphone app–delivered conversational agents. There is an urgent need for a robust evaluation of diverse health care conversational agents’ formats, focusing on their acceptability, safety, and effectiveness.

199 citations


Cites background or methods from "An Empathy-Driven, Conversational A..."

  • ...Eight studies presented conversational agents for managing mental health using techniques such as counseling [67]; cognitive behavioral therapy (CBT) [64,80] method of levels therapy [57]; positive psychology [61]; provision of a virtual companion [66]; and a combination of modalities such as CBT with mindfulness-based therapy, emotionally focused therapy, and motivational interviewing [75,81]....

    [...]

  • ...A variety of study designs were used in the included studies, comprising 20 case studies [44,48,51,61-63,66,69,71, 73-79,82,84,85,89], 4 surveys [55,56,59,65], 3 observational studies [53,86,87], 11 randomized controlled trials [46,49,50,57,64,67,72,80,81,83,88], 3 diagnostic accuracy studies [58,60,68], 3 controlled before and after studies [30,45,70], 2 ongoing trials [51,54], and 1 pilot study [47] (Figure 3)....

    [...]

  • ...agents spanned several different themes (Multimedia Appendices 3 and 4 [30,44-89])....

    [...]

  • ...A total of 22 studies were from European countries, including Italy [44,45], Switzerland [30,46-52], France [53,54], Portugal [55], The Netherlands [56], the United Kingdom [57-61], Spain [62,63], and Sweden [64]....

    [...]

  • ...The types of literature included 25 journal articles [44,48,55-57,61-65,67,69,72,74-76,80-87,89], 11 conference abstracts [45,47,49,50,52,59,70,71,73,78,79], 4 conference papers [30,46,66,77], 1 poster abstract [68], 4 electronic preprints [53,58,60,88], and 2 clinical trial protocols [51,54]....

    [...]

Journal ArticleDOI
TL;DR: An overview of the features of chatbots used by individuals for their mental health as reported in the empirical literature is provided to help guide potential users to the most appropriate chatbot to support theirmental health needs.

181 citations

Journal ArticleDOI
TL;DR: Chatbots may have a beneficial role to play in health care to support, motivate, and coach patients as well as for streamlining organizational tasks; in essence, chatbots could become a surrogate for nonmedical caregivers.
Abstract: Background: Many potential benefits for the uses of chatbots within the context of health care have been theorized, such as improved patient education and treatment compliance. However, little is known about the perspectives of practicing medical physicians on the use of chatbots in health care, even though these individuals are the traditional benchmark of proper patient care. Objective: This study aimed to investigate the perceptions of physicians regarding the use of health care chatbots, including their benefits, challenges, and risks to patients. Methods: A total of 100 practicing physicians across the United States completed a Web-based, self-report survey to examine their opinions of chatbot technology in health care. Descriptive statistics and frequencies were used to examine the characteristics of participants. Results: A wide variety of positive and negative perspectives were reported on the use of health care chatbots, including the importance to patients for managing their own health and the benefits on physical, psychological, and behavioral health outcomes. More consistent agreement occurred with regard to administrative benefits associated with chatbots; many physicians believed that chatbots would be most beneficial for scheduling doctor appointments (78%, 78/100), locating health clinics (76%, 76/100), or providing medication information (71%, 71/100). Conversely, many physicians believed that chatbots cannot effectively care for all of the patients’ needs (76%, 76/100), cannot display human emotion (72%, 72/100), and cannot provide detailed diagnosis and treatment because of not knowing all of the personal factors associated with the patient (71%, 71/100). Many physicians also stated that health care chatbots could be a risk to patients if they self-diagnose too often (714%, 74/100) and do not accurately understand the diagnoses (74%, 74/100). Conclusions: Physicians believed in both costs and benefits associated with chatbots, depending on the logistics and specific roles of the technology. Chatbots may have a beneficial role to play in health care to support, motivate, and coach patients as well as for streamlining organizational tasks; in essence, chatbots could become a surrogate for nonmedical caregivers. However, concerns remain on the inability of chatbots to comprehend the emotional state of humans as well as in areas where expert medical knowledge and intelligence is required.

156 citations


Cites background from "An Empathy-Driven, Conversational A..."

  • ...Early research has demonstrated the benefits of using health care chatbots, such as helping with diagnostic decision support [15,16], promoting and increasing physical activity [17], and cognitive behavioral therapy for psychiatric and somatic disorders [18-24], which provide effective, acceptable, and practical health care with accuracy comparable with that of human physicians....

    [...]

  • ...Some research and viewpoints on health care chatbots have been published from international researchers around the world [13-27]; however, to the best of our knowledge, this was the first study to examine physicians’ perspectives on the direct use of chatbots in their practice....

    [...]

Journal ArticleDOI
TL;DR: The role of personalization in improving health outcomes was not assessed directly and most of the studies in this review implemented the personalization features without theoretical or evidence-based support for them and did not leverage the recent developments in other domains ofpersonalization.
Abstract: Background: The personalization of conversational agents with natural language user interfaces is seeing increasing use in health care applications, shaping the content, structure, or purpose of the dialogue between humans and conversational agents. Objective: The goal of this systematic review was to understand the ways in which personalization has been used with conversational agents in health care and characterize the methods of its implementation. Methods: We searched on PubMed, Embase, CINAHL, PsycInfo, and ACM Digital Library using a predefined search strategy. The studies were included if they: (1) were primary research studies that focused on consumers, caregivers, or health care professionals; (2) involved a conversational agent with an unconstrained natural language interface; (3) tested the system with human subjects; and (4) implemented personalization features. Results: The search found 1958 publications. After abstract and full-text screening, 13 studies were included in the review. Common examples of personalized content included feedback, daily health reports, alerts, warnings, and recommendations. The personalization features were implemented without a theoretical framework of customization and with limited evaluation of its impact. While conversational agents with personalization features were reported to improve user satisfaction, user engagement and dialogue quality, the role of personalization in improving health outcomes was not assessed directly. Conclusions: Most of the studies in our review implemented the personalization features without theoretical or evidence-based support for them and did not leverage the recent developments in other domains of personalization. Future research could incorporate personalization as a distinct design factor with a more careful consideration of its impact on health outcomes and its implications on patient safety, privacy, and decision-making.

138 citations


Cites background or methods from "An Empathy-Driven, Conversational A..."

  • ...Across all the studies, data explicitly entered by the users included personal goals [35,37,46,47], symptoms and medications [37,44,45], measurement of vital signs [39,40], knowledge level on a specific topic [38], and daily practices [38]....

    [...]

  • ...The purposes of providing personalized content and conversations were to: (1) improve user engagement [35,37,38] and dialogue quality [42,43,54]; (2) provide timely feedback [39,40], adaptive user support [41], and adaptive training [36,38]; and, (3) support self-reflection [45,46]....

    [...]

  • ...Information needed for personalization was provided explicitly by the users in seven studies [35,37-40,44,45], and obtained implicitly by the system in one study [36] where the conversational agent analyzed users’ audio-visual features, such as facial expression and head position, to determine its feedback....

    [...]

  • ...Wysa (Inkster et al, 2018) [45] • Significant reduction in depression scores in both high (P<....

    [...]

  • ...NR Wellbeing support app for users with Wysa (Inkster et al, 2018) [45] • • • Personalized conversational pathways based on a user’s interaction, messages, and context Individuated Explicit: User responses to built-in assessment questionsymptoms of depresnaire and emotions sion, aiming to build expressed in a written mental resilience conversation and promote mental wellbeing...

    [...]

References
More filters
Journal ArticleDOI
TL;DR: Thematic analysis is a poorly demarcated, rarely acknowledged, yet widely used qualitative analytic method within psychology as mentioned in this paper, and it offers an accessible and theoretically flexible approach to analysing qualitative data.
Abstract: Thematic analysis is a poorly demarcated, rarely acknowledged, yet widely used qualitative analytic method within psychology. In this paper, we argue that it offers an accessible and theoretically flexible approach to analysing qualitative data. We outline what thematic analysis is, locating it in relation to other qualitative analytic methods that search for themes or patterns, and in relation to different epistemological and ontological positions. We then provide clear guidelines to those wanting to start thematic analysis, or conduct it in a more deliberate and rigorous way, and consider potential pitfalls in conducting thematic analysis. Finally, we outline the disadvantages and advantages of thematic analysis. We conclude by advocating thematic analysis as a useful and flexible method for qualitative research in and beyond psychology.

103,789 citations


"An Empathy-Driven, Conversational A..." refers methods in this paper

  • ...A qualitative thematic analysis, as proposed by Braun and Clarke, 2006 [35,36], on in-app feedback responses was performed....

    [...]

  • ...Both studies processed qualitative data gathered from responses to open-ended questions at postmeasurement using thematic analysis (Braun and Clarke, 2006)....

    [...]

  • ...An analysis of users’ in-app feedback responses was performed using thematic analysis [35,36] to measure engagement effectiveness....

    [...]

Journal Article
TL;DR: The development and initial psychometric evaluation of the Resilience Scale in a sample of 810 community-dwelling older adults support the internal consistency reliability and concurrent validity of the RS as an instrument to measure resilience.
Abstract: This study describes the development and initial psychometric evaluation of the 25-item Resilience Scale (RS) in a sample of 810 community-dwelling older adults. Principal components factor analysis of the RS was conducted followed by oblimin rotation indicating that the factor structure represented two factors (Personal Competence and Acceptance of Self and Life). Positive correlations with adaptational outcomes (physical health, morale, and life satisfaction) and a negative correlation with depression supported concurrent validity of the RS. The results of this study support the internal consistency reliability and concurrent validity of the RS as an instrument to measure resilience.

3,122 citations


"An Empathy-Driven, Conversational A..." refers methods in this paper

  • ...Future work should deploy repeated measure questionnaires such as Resilience Scale RS-14 [42], which may be more sensitive to changes in resilience in the general population....

    [...]

Journal ArticleDOI
TL;DR: The field of health and wellbeing scholarship has a strong tradition of qualitative research* and rightly so, and rich and compelling insights into the real worlds, experiences, and perspectives of patients and health care professionals can be found through quantitative methods.
Abstract: The field of health and wellbeing scholarship has a strong tradition of qualitative research*and rightly so. Qualitative research offers rich and compelling insights into the real worlds, experiences, and perspectives of patients and health care professionals in ways that are completely different to, but also sometimes complimentary to, the knowledge we can obtain through quantitative methods. (Published: 16 October 2014) Citation: Int J Qualitative Stud Health Well-being 2014, 9 : 26152 - http://dx.doi.org/10.3402/qhw.v9.26152

1,590 citations

Journal ArticleDOI
TL;DR: Regression to the mean is a ubiquitous phenomenon in repeated data and should always be considered as a possible cause of an observed change and can be alleviated through better study design and use of suitable statistical methods.
Abstract: Background Regression to the mean (RTM) is a statistical phenomenon that can make natural variation in repeated data look like real change. It happens when unusually large or small measurements tend to be followed by measurements that are closer to the mean. Methods We give some examples of the phenomenon, and discuss methods to overcome it at the design and analysis stages of a study. Results The effect of RTM in a sample becomes more noticeable with increasing measurement error and when follow-up measurements are only examined on a sub-sample selected using a baseline value. Conclusions RTM is a ubiquitous phenomenon in repeated data and should always be considered as a possible cause of an observed change. Its effect can be alleviated through better study design and use of suitable statistical methods.

1,500 citations


"An Empathy-Driven, Conversational A..." refers background in this paper

  • ...The authors expected that regression to the mean (whereby values that are initially measured as extreme are more likely to be moderate on subsequent measurement) might play a role in this apparent large improvement [40]....

    [...]

Journal ArticleDOI
TL;DR: This work outlines why AUC is the preferred measure of predictive or diagnostic accuracy in forensic psychology or psychiatry, and urges researchers and practitioners to use numbers rather than verbal labels to characterize effect sizes.
Abstract: In order to facilitate comparisons across follow-up studies that have used different measures of effect size, we provide a table of effect size equivalencies for the three most common measures: ROC area (AUC), Cohen's d, and r. We outline why AUC is the preferred measure of predictive or diagnostic accuracy in forensic psychology or psychiatry, and we urge researchers and practitioners to use numbers rather than verbal labels to characterize effect sizes.

1,363 citations


"An Empathy-Driven, Conversational A..." refers background in this paper

  • ...The CL gives the probability that a user picked at random from the high users group will have a higher average improvement than a user picked at random from the low users group [38,39]....

    [...]