Developing and Evaluating Digital Interventions to Promote Behavior Change in Health and Health Care: Recommendations Resulting From an International Workshop
TL;DR: A key conclusion was that sustained engagement is not always required and that for each intervention it is useful to establish what constitutes “effective engagement,” that is, sufficient engagement to achieve the intended outcomes.
Abstract: Devices and programs using digital technology to foster or support behavior change (digital interventions) are increasingly ubiquitous, being adopted for use in patient diagnosis and treatment, self-management of chronic diseases, and in primary prevention. They have been heralded as potentially revolutionizing the ways in which individuals can monitor and improve their health behaviors and health care by improving outcomes, reducing costs, and improving the patient experience. However, we are still mainly in the age of promise rather than delivery. Developing and evaluating these digital interventions presents new challenges and new versions of old challenges that require use of improved and perhaps entirely new methods for research and evaluation. This article discusses these challenges and provides recommendations aimed at accelerating the rate of progress in digital behavior intervention research and practice. Areas addressed include intervention development in a rapidly changing technological landscape, promoting user engagement, advancing the underpinning science and theory, evaluating effectiveness and cost-effectiveness, and addressing issues of regulatory, ethical, and information governance. This article is the result of a two-day international workshop on how to create, evaluate, and implement effective digital interventions in relation to health behaviors. It was held in London in September 2015 and was supported by the United Kingdom's Medical Research Council (MRC), the National Institute for Health Research (NIHR), the Methodology Research Programme (PI Susan Michie), and the Robert Wood Johnson Foundation of the United States (PI Kevin Patrick). Important recommendations to manage the rapid pace of change include considering using emerging techniques from data science, machine learning, and Bayesian approaches and learning from other disciplines including computer science and engineering. With regard to assessing and promoting engagement, a key conclusion was that sustained engagement is not always required and that for each intervention it is useful to establish what constitutes "effective engagement," that is, sufficient engagement to achieve the intended outcomes. The potential of digital interventions for testing and advancing theories of behavior change by generating ecologically valid, real-time objective data was recognized. Evaluations should include all phases of the development cycle, designed for generalizability, and consider new experimental designs to make the best use of rich data streams. Future health economics analyses need to recognize and model the complex and potentially far-reaching costs and benefits of digital interventions. In terms of governance, developers of digital behavior interventions should comply with existing regulatory frameworks, but with consideration for emerging standards around information governance, ethics, and interoperability.
Citations
More filters
TL;DR: An overview of approaches to intervention development can help researchers to understanding the variety of existing approaches, and to understand the range of possible actions involved in intervention development, prior to assessing feasibility or piloting the intervention.
Abstract: Interventions need to be developed prior to the feasibility and piloting phase of a study. There are a variety of published approaches to developing interventions, programmes or innovations to improve health. Identifying different types of approach, and synthesising the range of actions taken within this endeavour, can inform future intervention development. This study is a systematic methods overview of approaches to intervention development. Approaches were considered for inclusion if they described how to develop or adapt an intervention in a book, website or journal article published after 2007, or were cited in a primary research study reporting the development of a specific intervention published in 2015 or 2016. Approaches were read, a taxonomy of approaches was developed and the range of actions taken across different approaches were synthesised. Eight categories of approach to intervention development were identified. (1) Partnership, where people who will use the intervention participate equally with the research team in decision-making about the intervention throughout the development process. (2) Target population-centred, where the intervention is based on the views and actions of the people who will use it. (3) Evidence and theory-based, where the intervention is based on published research evidence and existing theories. (4) Implementation-based, where the intervention is developed with attention to ensuring it will be used in the real world. (5) Efficiency-based, where components of an intervention are tested using experimental designs to select components which will optimise efficiency. (6) Stepped or phased, where interventions are developed with an emphasis on following a systematic set of processes. (7) Intervention-specific, where an approach is constructed for a specific type of intervention. (8) Combination, where existing approaches to intervention development are formally combined. The actions from approaches in all eight categories were synthesised to identify 18 actions to consider when developing interventions. This overview of approaches to intervention development can help researchers to understand the variety of existing approaches, and to understand the range of possible actions involved in intervention development, prior to assessing feasibility or piloting the intervention. Findings from this overview will contribute to future guidance on intervention development. PROSPERO CRD42017080553
.
234 citations
Additional excerpts
...future rather than current guidance [23]....
[...]
TL;DR: It can be concluded that adherence to eHealth technology is an underdeveloped and often improperly used concept in the existing body of literature.
Abstract: Background: In electronic health (eHealth) evaluations, there is increasing attention for studying the actual usage of a technology in relation to the outcomes found, often by studying the adherence to the technology. On the basis of the definition of adherence, we suggest that the following three elements are necessary to determine adherence to eHealth technology: (1) the ability to measure the usage behavior of individuals; (2) an operationalization of intended use; and (3) an empirical, theoretical, or rational justification of the intended use. However, to date, little is known on how to operationalize the intended usage of and the adherence to different types of eHealth technology. Objective: The study aimed to improve eHealth evaluations by gaining insight into when, how, and by whom the concept of adherence has been used in previous eHealth evaluations and finding a concise way to operationalize adherence to and intended use of different eHealth technologies. Methods: A systematic review of eHealth evaluations was conducted to gain insight into how the use of the technology was measured, how adherence to different types of technologies was operationalized, and if and how the intended use of the technology was justified. Differences in variables between the use of the technology and the operationalization of adherence were calculated using a chi-square test of independence. Results: In total, 62 studies were included in this review. In 34 studies, adherence was operationalized as “the more use, the better,” whereas 28 studies described a threshold for intended use of the technology as well. Out of these 28, only 6 reported a justification for the intended use. The proportion of evaluations of mental health technologies reporting a justified operationalization of intended use is lagging behind compared with evaluations of lifestyle and chronic care technologies. The results indicated that a justification of intended use does not require extra measurements to determine adherence to the technology. Conclusions: The results of this review showed that to date, justifications for intended use are often missing in evaluations of adherence. Evidently, it is not always possible to estimate the intended use of a technology. However, such measures do not meet the definition of adherence and should therefore be referred to as the actual usage of the technology. Therefore, it can be concluded that adherence to eHealth technology is an underdeveloped and often improperly used concept in the existing body of literature. When defining the intended use of a technology and selecting valid measures for adherence, the goal or the assumed working mechanisms should be leading. Adherence can then be standardized, which will improve the comparison of adherence rates to different technologies with the same goal and will provide insight into how adherence to different elements contributed to the outcomes.
204 citations
Cites background from "Developing and Evaluating Digital I..."
...For every technology, a proportion of the users will not use the intervention at all, will stop using the technology after a period, or will not use the available elements of the technology as intended [1,6-8]....
[...]
...An important reason for the lack of justifications for the intended use of eHealth technologies might be that there is a lack of knowledge regarding the working mechanisms of technology-based applications [8]....
[...]
...showed that adherence is a multidimensional concept, influenced by a range of technological, environmental, and individual factors altogether that cannot be evaluated by technology usage alone [8,84]....
[...]
University of Adelaide1, Ghent University2, University of Limerick3, Central Queensland University4, University of South Australia5, Erasmus University Rotterdam6, University of Malaya7, National University of Singapore8, Coventry University9, University of Arizona10, Public Health Research Institute11
TL;DR: An overview of engagement measurement options that can be employed in eHealth and mHealth behavior change intervention evaluations, discuss methodological considerations, and provide direction for future research is provided.
Abstract: Engagement in electronic health (eHealth) and mobile health (mHealth) behavior change interventions is thought to be important for intervention effectiveness, though what constitutes engagement and how it enhances efficacy has been somewhat unclear in the literature Recently published detailed definitions and conceptual models of engagement have helped to build consensus around a definition of engagement and improve our understanding of how engagement may influence effectiveness This work has helped to establish a clearer research agenda However, to test the hypotheses generated by the conceptual modules, we need to know how to measure engagement in a valid and reliable way The aim of this viewpoint is to provide an overview of engagement measurement options that can be employed in eHealth and mHealth behavior change intervention evaluations, discuss methodological considerations, and provide direction for future research To identify measures, we used snowball sampling, starting from systematic reviews of engagement research as well as those utilized in studies known to the authors A wide range of methods to measure engagement were identified, including qualitative measures, self-report questionnaires, ecological momentary assessments, system usage data, sensor data, social media data, and psychophysiological measures Each measurement method is appraised and examples are provided to illustrate possible use in eHealth and mHealth behavior change research Recommendations for future research are provided, based on the limitations of current methods and the heavy reliance on system usage data as the sole assessment of engagement The validation and adoption of a wider range of engagement measurements and their thoughtful application to the study of engagement are encouraged
196 citations
TL;DR: This article presents an introduction to, and a systematic review of, current ML work regarding psycho-socially based mental health conditions from the computing and HCI literature, and reflects on the current state-of-the-art of ML work for mental health.
Abstract: High prevalence of mental illness and the need for effective mental health care, combined with recent advances in AI, has led to an increase in explorations of how the field of machine learning (ML) can assist in the detection, diagnosis and treatment of mental health problems. ML techniques can potentially offer new routes for learning patterns of human behavior; identifying mental health symptoms and risk factors; developing predictions about disease progression; and personalizing and optimizing therapies. Despite the potential opportunities for using ML within mental health, this is an emerging research area, and the development of effective ML-enabled applications that are implementable in practice is bound up with an array of complex, interwoven challenges. Aiming to guide future research and identify new directions for advancing development in this important domain, this article presents an introduction to, and a systematic review of, current ML work regarding psycho-socially based mental health conditions from the computing and HCI literature. A quantitative synthesis and qualitative narrative review of 54 papers that were included in the analysis surfaced common trends, gaps, and challenges in this space. Discussing our findings, we (i) reflect on the current state-of-the-art of ML work for mental health, (ii) provide concrete suggestions for a stronger integration of human-centered and multi-disciplinary approaches in research and development, and (iii) invite more consideration of the potentially far-reaching personal, social, and ethical implications that ML models and interventions can have, if they are to find widespread, successful adoption in real-world mental health contexts.
153 citations
[...]
TL;DR: It is argued that while self-tracking may sometimes prove to be an adequate method to shed light on particular aspects of oneself and can be used to strengthen one’s autonomy, self- tracking technologies often cancel out these benefits by exposing too much about oneself to an unspecified audience, thus undermining the informational privacy boundaries necessary for living an autonomous life.
Abstract: This paper critically engages with new self-tracking technologies. In particular, it focuses on a conceptual tension between the idea that disclosing personal information increases one's autonomy and the idea that informational privacy is a condition for autonomous personhood. I argue that while self-tracking may sometimes prove to be an adequate method to shed light on particular aspects of oneself and can be used to strengthen one's autonomy, self-tracking technologies often cancel out these benefits by exposing too much about oneself to an unspecified audience, thus undermining the informational privacy boundaries necessary for living an autonomous life.
143 citations
References
More filters
TL;DR: A checklist instrument that has the potential to improve reporting and provides a basis for evaluating the validity and applicability of eHealth trials is developed.
Abstract: Background: Web-based and mobile health interventions (also called “Internet interventions” or "eHealth/mHealth interventions") are tools or treatments, typically behaviorally based, that are operationalized and transformed for delivery via the Internet or mobile platforms. These include electronic tools for patients, informal caregivers, healthy consumers, and health care providers. The Consolidated Standards of Reporting Trials (CONSORT) statement was developed to improve the suboptimal reporting of randomized controlled trials (RCTs). While the CONSORT statement can be applied to provide broad guidance on how eHealth and mHealth trials should be reported, RCTs of web-based interventions pose very specific issues and challenges, in particular related to reporting sufficient details of the intervention to allow replication and theory-building. Objective: To develop a checklist, dubbed CONSORT-EHEALTH (Consolidated Standards of Reporting Trials of Electronic and Mobile HEalth Applications and onLine TeleHealth), as an extension of the CONSORT statement that provides guidance for authors of eHealth and mHealth interventions. Methods: A literature review was conducted, followed by a survey among eHealth experts and a workshop. Results: A checklist instrument was constructed as an extension of the CONSORT statement. The instrument has been adopted by the Journal of Medical Internet Research (JMIR) and authors of eHealth RCTs are required to submit an electronic checklist explaining how they addressed each subitem. Conclusions: CONSORT-EHEALTH has the potential to improve reporting and provides a basis for evaluating the validity and applicability of eHealth trials. Subitems describing how the intervention should be reported can also be used for non-RCT evaluation reports. As part of the development process, an evaluation component is essential; therefore, feedback from authors will be solicited, and a before-after study will evaluate whether reporting has been improved.
1,242 citations
TL;DR: A holistic framework is composed based on a participatory development approach, persuasive design techniques, and business modeling that serves as an evidence-based roadmap to demonstrate the impact of eHealth technologies more effectively.
Abstract: Background: Many eHealth technologies are not successful in realizing sustainable innovations in health care practices. One of the reasons for this is that the current development of eHealth technology often disregards the interdependencies between technology, human characteristics, and the socioeconomic environment, resulting in technology that has a low impact in health care practices. To overcome the hurdles with eHealth design and implementation, a new, holistic approach to the development of eHealth technologies is needed, one that takes into account the complexity of health care and the rituals and habits of patients and other stakeholders. Objective: The aim of this viewpoint paper is to improve the uptake and impact of eHealth technologies by advocating a holistic approach toward their development and eventual integration in the health sector. Methods: To identify the potential and limitations of current eHealth frameworks (1999–2009), we carried out a literature search in the following electronic databases: PubMed, ScienceDirect, Web of Knowledge, PiCarta, and Google Scholar. Of the 60 papers that were identified, 44 were selected for full review. We excluded those papers that did not describe hands-on guidelines or quality criteria for the design, implementation, and evaluation of eHealth technologies (28 papers). From the results retrieved, we identified 16 eHealth frameworks that matched the inclusion criteria. The outcomes were used to posit strategies and principles for a holistic approach toward the development of eHealth technologies; these principles underpin our holistic eHealth framework. Results: A total of 16 frameworks qualified for a final analysis, based on their theoretical backgrounds and visions on eHealth, and the strategies and conditions for the research and development of eHealth technologies. Despite their potential, the relationship between the visions on eHealth, proposed strategies, and research methods is obscure, perhaps due to a rather conceptual approach that focuses on the rationale behind the frameworks rather than on practical guidelines. In addition, the Web 2.0 technologies that call for a more stakeholder-driven approach are beyond the scope of current frameworks. To overcome these limitations, we composed a holistic framework based on a participatory development approach, persuasive design techniques, and business modeling. Conclusions: To demonstrate the impact of eHealth technologies more effectively, a fresh way of thinking is required about how technology can be used to innovate health care. It also requires new concepts and instruments to develop and implement technologies in practice. The proposed framework serves as an evidence-based roadmap.
883 citations
"Developing and Evaluating Digital I..." refers background in this paper
...These qualitative approaches are central to participatory user-centered design, which is the key to developing and evaluating DBCIs in order to ensure that they are engaging and effective [16,17]....
[...]
...To promote engagement with DBCIs, a “user-centered” [16] or “person-based” [17] approach is essential to ensure that interventions are responsive to users’ needs and preferences....
[...]
TL;DR: How to implement the person-based approach to digital interventions is described, illustrating the process with examples of the insights gained from the experience of carrying out over a thousand interviews with users, while developing public health and illness management interventions that have proven effective in trials involving tens of thousands of users.
Abstract: This paper describes an approach that we have evolved for developing successful digital interventions to help people manage their health or illness. We refer to this as the “person-based” approach to highlight the focus on understanding and accommodating the perspectives of the people who will use the intervention. While all intervention designers seek to elicit and incorporate the views of target users in a variety of ways, the person-based approach offers a distinctive and systematic means of addressing the user experience of intended behavior change techniques in particular and can enhance the use of theory-based and evidence-based approaches to intervention development. There are two key elements to the person-based approach. The first is a developmental process involving qualitative research with a wide range of people from the target user populations, carried out at every stage of intervention development, from planning to feasibility testing and implementation. This process goes beyond assessing acceptability, usability, and satisfaction, allowing the intervention designers to build a deep understanding of the psychosocial context of users and their views of the behavioral elements of the intervention. Insights from this process can be used to anticipate and interpret intervention usage and outcomes, and most importantly to modify the intervention to make it more persuasive, feasible, and relevant to users. The second element of the person-based approach is to identify “guiding principles” that can inspire and inform the intervention development by highlighting the distinctive ways that the intervention will address key context-specific behavioral issues. This paper describes how to implement the person-based approach, illustrating the process with examples of the insights gained from our experience of carrying out over a thousand interviews with users, while developing public health and illness management interventions that have proven effective in trials involving tens of thousands of users.
812 citations
"Developing and Evaluating Digital I..." refers background in this paper
...These qualitative approaches are central to participatory user-centered design, which is the key to developing and evaluating DBCIs in order to ensure that they are engaging and effective [16,17]....
[...]
...To promote engagement with DBCIs, a “user-centered” [16] or “person-based” [17] approach is essential to ensure that interventions are responsive to users’ needs and preferences....
[...]
TL;DR: In this article, the impact of guidance on the efficacy of Internet-based interventions was systematically reviewed and a systematic search of MEDLINE, CENTRAL and PsycINFO, PsycARTICLES and Psyndex (search date 4th June 2013) was conducted.
753 citations
"Developing and Evaluating Digital I..." refers background in this paper
...Adding human support is also known to promote engagement with many interventions [18]....
[...]
Book•
31 Jul 2015
TL;DR: This book provides an introduction to the field of applied ontology that is of particular relevance to biomedicine, covering theoretical components of ontologies, best practices for ontology design, and examples of biomedical ontologies in use.
Abstract: In the era of "big data," science is increasingly information driven, and the potential for computers to store, manage, and integrate massive amounts of data has given rise to such new disciplinary fields as biomedical informatics. Applied ontology offers a strategy for the organization of scientific information in computer-tractable form, drawing on concepts not only from computer and information science but also from linguistics, logic, and philosophy. This book provides an introduction to the field of applied ontology that is of particular relevance to biomedicine, covering theoretical components of ontologies, best practices for ontology design, and examples of biomedical ontologies in use.After defining an ontology as a representation of the types of entities in a given domain, the book distinguishes between different kinds of ontologies and taxonomies, and shows how applied ontology draws on more traditional ideas from metaphysics. It presents the core features of the Basic Formal Ontology (BFO), now used by over one hundred ontology projects around the world, and offers examples of domain ontologies that utilize BFO. The book also describes Web Ontology Language (OWL), a common framework for Semantic Web technologies. Throughout, the book provides concrete recommendations for the design and construction of domain ontologies.
659 citations
Additional excerpts
...Such knowledge-organizing structures are called “ontologies” [22,23]....
[...]