Synthesis of multiple randomized controlled trials (RCTs) in a systematic review can summarize the effects of individual outcomes and provide numerical answers about the effectiveness of interventions. Filtering of searches is time consuming, and no single method fulfills the principal requirements of speed with accuracy. Automation of systematic reviews is driven by a necessity to expedite the availability of current best evidence for policy and clinical decision-making. We developed Rayyan (
 http://rayyan.qcri.org
 ), a free web and mobile app, that helps expedite the initial screening of abstracts and titles using a process of semi-automation while incorporating a high level of usability. For the beta testing phase, we used two published Cochrane reviews in which included studies had been selected manually. Their searches, with 1030 records and 273 records, were uploaded to Rayyan. Different features of Rayyan were tested using these two reviews. We also conducted a survey of Rayyan’s users and collected feedback through a built-in feature. Pilot testing of Rayyan focused on usability, accuracy against manual methods, and the added value of the prediction feature. The “taster” review (273 records) allowed a quick overview of Rayyan for early comments on usability. The second review (1030 records) required several iterations to identify the previously identified 11 trials. The “suggestions” and “hints,” based on the “prediction model,” appeared as testing progressed beyond five included studies. Post rollout user experiences and a reflexive response by the developers enabled real-time modifications and improvements. The survey respondents reported 40% average time savings when using Rayyan compared to others tools, with 34% of the respondents reporting more than 50% time savings. In addition, around 75% of the respondents mentioned that screening and labeling studies as well as collaborating on reviews to be the two most important features of Rayyan. As of November 2016, Rayyan users exceed 2000 from over 60 countries conducting hundreds of reviews totaling more than 1.6M citations. Feedback from users, obtained mostly through the app web site and a recent survey, has highlighted the ease in exploration of searches, the time saved, and simplicity in sharing and comparing include-exclude decisions. The strongest features of the app, identified and reported in user feedback, were its ability to help in screening and collaboration as well as the time savings it affords to users. Rayyan is responsive and intuitive in use with significant potential to lighten the load of reviewers.

/pdf/rayyan-a-web-and-mobile-app-for-systematic-reviews-4xu8f0bave.pdf

Rayyan-a web and mobile app for systematic reviews.

Cognitive engineering

This paper describes an approach that we have evolved for developing successful digital interventions to help people manage their health or illness. We refer to this as the “person-based” approach to highlight the focus on understanding and accommodating the perspectives of the people who will use the intervention. While all intervention designers seek to elicit and incorporate the views of target users in a variety of ways, the person-based approach offers a distinctive and systematic means of addressing the user experience of intended behavior change techniques in particular and can enhance the use of theory-based and evidence-based approaches to intervention development. There are two key elements to the person-based approach. The first is a developmental process involving qualitative research with a wide range of people from the target user populations, carried out at every stage of intervention development, from planning to feasibility testing and implementation. This process goes beyond assessing acceptability, usability, and satisfaction, allowing the intervention designers to build a deep understanding of the psychosocial context of users and their views of the behavioral elements of the intervention. Insights from this process can be used to anticipate and interpret intervention usage and outcomes, and most importantly to modify the intervention to make it more persuasive, feasible, and relevant to users. The second element of the person-based approach is to identify “guiding principles” that can inspire and inform the intervention development by highlighting the distinctive ways that the intervention will address key context-specific behavioral issues. This paper describes how to implement the person-based approach, illustrating the process with examples of the insights gained from our experience of carrying out over a thousand interviews with users, while developing public health and illness management interventions that have proven effective in trials involving tens of thousands of users.

The Person-Based Approach to Intervention Development: Application to Digital Health-Related Behavior Change Interventions

https://www.healthit.gov/sites/default/files/archive/HIT%20Standards%20Committee/2013/Implementation/2013-07-23/jammedinformassoc-2013-middleton_0.pdf

Enhancing patient safety and quality of care by improving the usability of electronic health record systems: recommendations from AMIA

The large and growing number of published studies, and their increasing rate of publication, makes the task of identifying relevant studies in an unbiased way for inclusion in systematic reviews both complex and time consuming. Text mining has been offered as a potential solution: through automating some of the screening process, reviewer time can be saved. The evidence base around the use of text mining for screening has not yet been pulled together systematically; this systematic review fills that research gap. Focusing mainly on non-technical issues, the review aims to increase awareness of the potential of these technologies and promote further collaborative research between the computer science and systematic review communities. Five research questions led our review: what is the state of the evidence base; how has workload reduction been evaluated; what are the purposes of semi-automation and how effective are they; how have key contextual problems of applying text mining to the systematic review field been addressed; and what challenges to implementation have emerged? We answered these questions using standard systematic review methods: systematic and exhaustive searching, quality-assured data extraction and a narrative synthesis to synthesise findings. The evidence base is active and diverse; there is almost no replication between studies or collaboration between research teams and, whilst it is difficult to establish any overall conclusions about best approaches, it is clear that efficiencies and reductions in workload are potentially achievable. On the whole, most suggested that a saving in workload of between 30% and 70% might be possible, though sometimes the saving in workload is accompanied by the loss of 5% of relevant studies (i.e. a 95% recall). Using text mining to prioritise the order in which items are screened should be considered safe and ready for use in ‘live’ reviews. The use of text mining as a ‘second screener’ may also be used cautiously. The use of text mining to eliminate studies automatically should be considered promising, but not yet fully proven. In highly technical/clinical areas, it may be used with a high degree of confidence; but more developmental and evaluative work is needed in other disciplines.

/pdf/using-text-mining-for-study-identification-in-systematic-gm1ndl5pf0.pdf

Using text mining for study identification in systematic reviews: a systematic review of current approaches

Review of health information technology usability study methodologies

Reducing workload in systematic review preparation using automated citation classification.

Assessment of the Health IT Usability Evaluation Model (Health-ITUEM) for evaluating mobile health (mHealth) technology

Time motion studies in healthcare

We developed a customizable questionnaire, the Health Information Technology (IT) Usability Evaluation Scale (Health-ITUES) and conducted an exploratory factor analysis to examine the scale's psychometric properties. Nurses (n=377) completed Health-ITUES to rate the usability of a web-based communication system for scheduling nursing staff. The analysis revealed a four-factor structure of Health-ITUES. The results provided preliminary evidence for the factorial validity and internal consistency reliability of Health-ITUES.

Po-Yin Yen

Papers

Review of health information technology usability study methodologies

Reducing workload in systematic review preparation using automated citation classification.

Assessment of the Health IT Usability Evaluation Model (Health-ITUEM) for evaluating mobile health (mHealth) technology

Time motion studies in healthcare

Development of a Customizable Health IT Usability Evaluation Scale.