Rayyan-a web and mobile app for systematic reviews.

doi:10.1186/S13643-016-0384-4

Ouzzani et al. Systematic Reviews

(2016) 5:210

DOI 10.1186/s13643-016-0384-4

METHODOLOGY Open Access

Rayyan—a web and mobile app for

systematic reviews

Mourad Ouzzani

1*

, Hossam Hammady

1

, Zbys Fedorowicz

2

and Ahmed Elmagarmid

1

Abstract

Background: Synthesis of multiple randomized controlled trials (RCTs) in a systematic review can summarize the

effects of individual outcomes and provide numerical answers about the effectiveness of interventions. Filtering of

searches is time consuming, and no single method fulfills the principal requirements of speed with accuracy.

Automation of systematic reviews is driven by a necessity to expedite the availability of current best evidence for

policy and clinical decision-making.

We developed Rayyan (http://rayyan.qcri.org), a free web and mobile app, that helps expedite the initial screening of

abstracts and titles using a process of semi-automation while incorporating a high level of usability. For the beta

testing phase, we used two published Cochrane reviews in which included studies had been selected manually.

Their searches, with 1030 records and 273 records, were uploaded to Rayyan. Different features of Rayyan were tested

using these two reviews. We also conducted a survey of Rayyan’s users and collected feedback through a built-in

feature.

Results: Pilot testing of Rayyan focused on usability, accuracy against manual methods, and the added value of the

prediction feature. The “taster” review (273 records) allowed a quick overview of Rayyan for early comments on

usability. The second review (1030 records) required several iterations to identify the previously identified 11 trials. The

“suggestions” and “hints,” based on the “prediction model,” appeared as testing progressed beyond five included

studies. Post rollout user experiences and a reflexive response by the developers enabled real-time modifications and

improvements. The survey respondents reported 40% average time savings when using Rayyan compared to others

tools, with 34% of the respondents reporting more than 50% time savings. In addition, around 75% of the

respondents mentioned that screening and labeling studies as well as collaborating on reviews to be the two most

important features of Rayyan.

As of November 2016, Rayyan users exceed 2000 from over 60 countries conducting hundreds of reviews totaling

more than 1.6M citations. Feedback from users, obtained mostly through the app web site and a recent survey, has

highlighted the ease in exploration of searches, the time saved, and simplicity in sharing and comparing

include-exclude decisions. The strongest features of the app, identified and reported in user feedback, were its ability

to help in screening and collaboration as well as the time savings it affords to users.

Conclusions: Rayyan is responsive and intuitive in use with significant potential to lighten the load of reviewers.

Keywords: Systematic reviews, Evidence-based medicine, Automation

*Correspondence: mouzzani@qf.org.qa

1

Qatar Computing Research Institute, HBKU, Doha, Qatar

Full list of author information is available at the end of the article

International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and

reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the

Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver

(http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.

Ouzzani et al. Systematic Reviews

(2016) 5:210

Page 2 of 10

Background

Randomized controlled trials (RCTs) play a pivotal role in

medical research and are widely considered to be the best

way of achieving results that can genuinely increase our

knowledge about treatment effectiveness [1]. Although

there is an increasing requirement for randomized con-

trolled trials to guide healthcare decision-making, the

synthesis of the results of more than one RCT in a system-

atic review can summarize the effects of their individual

outcomes and provide numerical answers about the effec-

tiveness of a particular intervention.

A systematic review is a summary of the medical liter-

ature that uses explicit methods to systematically search,

critically appraise, and synthesize the data on a specific

topic. The need for rigor in the production of systematic

reviews has led to the development of a formal process for

their conduct. This process has clearly designated steps

to identify primary studies and the methods which will

be employed to assess their methodological quality, the

way in which data will be extracted, and the statistical

techniques that will be used in the synthesis and report-

ing of that data [2]. Transparency and reproducibility are

assured through the documenting of all of the decisions

taken to include or exclude studies throughout the review

process.

Identification of studies: The overarching aim is to

ensure that an exhaustive scrutiny of the literature cre-

ates as comprehensive a list as possible of published and

unpublished primary studies which are deemed relevant

to answering the research question.

The number of citations generated by this search for

eligible studies will depend on a variety of factors not

least of all those involving some of the inherent aspects

of the clinical topic. Thus, a clinical intervention which

has been used extensively over a long period of time may

be underpinned by a large body of research which in

many instances may contain a substantial number of stud-

ies some of which may date back in excess of 20 years.

Other possible contributory factors will include the com-

parative “interest” in the topic by clinicians, healthcare

policy makers, and the media and may even include

the potentially “vested” interest of the pharmaceutical

industry.

Although the initial searches for trials for a systematic

review may in some cases identify up to, and possibly

extend beyond, 1000 citations, this will depend in part on

the level of sensitivity and specificity built into the search

strategy used to search the individual databases. While

it is difficult to generalize what number of references to

studies might be expected in an average yield, a minimum

of 100 would not be an unreasonable number for many

clinical topics.

Identification of potentially eligible studies: One of the

most time consuming aspects of conducting a systematic

review is the preliminary filtering or sifting through the

citations from the searches, particularly if these number

in the several hundreds and possibly in the thousands.

Systematic review authors use a variety of electronic or

manual methods to complete this task, which in any event

must be double-checked by a co-author to ensure that

all potentially eligible studies and those that require fur-

ther full-text assessment have been identified. In addition,

the tracking of decisions to include or exclude stud-

ies and the reporting of these judgments in a PRISMA

flow diagram is mandatory for all Cochrane reviews

and is now being done increasingly in other system-

atic reviews as this becomes a more widely accepted

prerequisite for manuscript publication [3]. Moreover,

the comprehensive documentation of these decisions

by the review authors ensures the transparency, clarity,

and traceability of the selection process and ultimately

reinforces the robustness of the completed systematic

review.

Identification and selection of studies can be challeng-

ing and very tedious, and a number of methods are used

by review authors to facilitate the process. This can be

performed either manually, i.e., by simply “highlighting”

them in the printed copy of the search document by the

use of different colors of a text marker, or electronically

using the text highlighting function in the electronic copy

of the search document. Alternative methods include the

use of software such as EndNote or Reference Manager, if

they are available to the review author. No single method

can satisfactorily fulfill all the principal requirements of

speed, accuracy, and simplicity in use, and each has its

advantages, disadvantages, and adherents.

Interest in the automation of systematic reviews has

been driven by a necessity to expedite the availability of

current best evidence for policy and clinical decision-

making as much as engaging with technology to allow

review authors to redirect their focus on aspects where

they are best at [4]. An increasing number of projects are

underway which focus on the automation of segments of

the systematic review process, and although several tools

and software have been developed, so far, none of them

span the entire process of review production [5].

Although the challenges faced by developers to auto-

mate and integrate the multiple steps in the workflow

may seem insurmountable, recent advances in technology

have helped overcome some of these hurdles [6]. However,

accuracy and efficiency should not be sacrificed at the

expense of speed, but flexibility, aligned with the poten-

tial for individual user customizability, should be built into

the tool to allow for a range of users to create and use

different personal preference-based interfaces. Automa-

tion should also target several key areas such as exploring

ways of enhancing the user interface and user experience,

developing systems which will ensure adequate workflow

Ouzzani et al. Systematic Reviews

(2016) 5:210

Page 3 of 10

support, and the fostering of further developments in

machine learning and data/text mining.

The process of automation of systematic reviews con-

tinues to present a number of additional challenges in that

many of the tools have been developed independently as

stand-alone software and are often not compatible with

other tools [5]. In some instances, appropriate reliability

and functionality testing has not been undertaken, and

some tools are no longer being maintained by the devel-

opers or are prohibitively expensive to the average user.

Moreover, some of the tools currently available require

a level of technical skills beyond that of many review

authors and also involve a steep learning curve and level

of complexity which may necessitate a repetitive learn/re-

learn phase if they are not used regularly. All of these

challenges show how unsatisfying the existing landscape

for systematic review automation is. The developers of

Rayyan aim to address these challenges for providing an

integrated solution, by working directly with systematic

reviewers whilst continuously taking into account users’

feedback.

Objectives

Rayyan (http://rayyan.qcri.org) was developed specifically

to expedite the initial screening of abstracts and titles

using a process of semi-automation but with a clear

objective of incorporating a level of usability which would

be compatible with the skillset of a broad cross-section

of potential users. The ab initio objectives of the develop-

ers of the Rayyan app were to try and circumvent some of

the complexities and challenges faced by reviewers with

some of the existing tools. While our ultimate goal is to

support the entire systematic review process, we initially

focus on facilitating abstract/title screening and collabora-

tion in addition to other supporting features around them.

Thus,muchofthefocusofthedevelopmentwasoncreat-

ing an inbuilt user-definable and partly self-customizable

interface which would ensure Rayyan was largely intuitive

in use as well as being user-friendly at all skill levels. We

present here an exceptional case report of the develop-

ment process of Rayyan, an app for the rapid exploring

and filtering of searches for eligible studies for systematic

reviews.

Methods

There was a recognition by the developers of the need for

a tool which would satisfy the requirements of a broad

spectrum of review authors with a diverse range of com-

petencies and skills and specifically one which would

permit rapid and reliable exploration and sharing of search

results but without being technologically burdensome.

Therefore, engagement with an experienced Cochrane

systematic review author (ZF) who had worked exten-

sively with a large number of co-authors with mixed levels

of experience proved to be pivotal to the development

process.

The app underwent pilot testing prior to release and

had extensive subsequent evaluation from a wide range of

users, with a variety of skill levels and competencies, from

across the globe. Sharing of user experiences and a reflex-

ive response by the developers to an evolving “wish list”

of requests by users enabled modifications and improve-

ments to be made progressively and in real time, all of

which proved to be a highly productive and effective

collaboration in the development of Rayyan.

Overview and architecture

Rayyan is built on top of a cloud-based multi-tier service-

oriented elastic architecture (Fig. 1). Scalability in Rayyan

is underpinned by this cloud-based architecture which

allows it to scale accordingly during peak times and as

the number of users grows and they create more reviews

and upload more citations. Moreover, at times, Rayyan

may be actively processing data for tens of users or is

just staying idle. The cloud-based architecture enables it

to expand or shrink its hardware resources as needed. As

a result, it is cost effective in idle times, with no costs

incurred for resources not being used, and at the same

time horizontally scales out in busy times easily. Part of

the resources are only manually scalable, which means

that Rayyan administrators will need to upgrade them as

needed, for example, in increasing database storage needs,

push notifications volume, and email messages volume.

Other resources are automatically scalable to support the

appropriate traffic in a cost-effective manner without sac-

rificing performance. This applies to web servers and

background job workers.

Rayyan itself is written in the popular open-source

framework Ruby on Rails [7], and runs on Heroku [8]

which is a Platform as a Service based on the cloud-

hosting Amazon Web Services. It integrates with other

cloud services to fulfill the different tiers it requires.

Examples of these services are Heroku Postgres [9] for

SQL database management; Logentries [10] for central

logging, tagging, and alerting; NewRelic [11] for app ana-

lytics, health monitoring, and alerting; Pusher [12] for

real-time push notifications; and HireFire [13] for auto-

scaling the app according to load.

Workflow and user experience

After logging into Rayyan, users are presented with a

dashboard of all their current reviews (Fig. 2). They can

either create a new review or work on an existing one.

For each review, they upload one or more citation file

obtained from searching different databases. Rayyan sup-

ports several standard formats, e.g., RefMan RIS and

EndNote. At the outset, Rayyan processes the citation file

Ouzzani et al. Systematic Reviews

(2016) 5:210

Page 4 of 10

Fig. 1 Rayyan architecture. Rayyan is a fully cloud-based architecture that uses a cloud platform as a service allowing elastic scaling of resources as

we get more users and more requests. Rayyan’s workers are distributed using the load balancer to different app servers (Ruby web workers). These

workers are elastic; they auto-scale based on traffic to guarantee minimal response time. For longer jobs or the elastic delayed jobs (the worker

bees), such as upload parsing, similarity computation, and label predictions, they are handled through a queuing system. All workers have access to

the storage layers: Postgres (for permanent storage), Solr (for indexing and searching), and Memcached (for caching results). Other parts of Rayyan,

written in Java, are attachable to the jobs using an Apache Thrift service. Real-time notifications, on job completion or chat messages, for example,

are delivered using Pusher, while other transactional information are delivered using the Mailchimp Mandrill service. All system activities are logged

by Logentries and later backed up on AWS S3, while live instrumentation and monitoring is done by NewRelic

by extracting different metadata, e.g., title, authors, and

computing others, e.g., MeSH terms and language of the

article, for each article or study in the citation file. These

will then populate the facets in the review workbench

(Fig. 3) to help explore and filter the studies. MeSH terms

are presented as a word cloud allowing users to quickly

grasp the main topics presented in the studies. In addi-

tion, users can filter studies based on two predefined lists

of keywords that will most likely hint to either include or

excludeastudy.Theusercanalsomodifythesetwolists

by removing and adding keywords, thus giving more flex-

ibility in the labeling and selection of studies. Rayyan was

seeded with two lists obtained from the EMBASE project

to filter RCTs [14].

Users can also label their citations and define their indi-

vidual reasons for exclusion which facilitates the sharing

and tracking of these decisions. Citations can be explored

through a similarity graph (Fig. 4) in which the citations

are represented as nodes in a graph and clustered based

on how similar they are (using an edit distance) in terms

of title and abstract content as well as common authors.

The similarity thresholds can be tuned independently for

each attribute, i.e., title, abstract, and authors, as well an

overall threshold.

Rayyan mobile app

With the mobile app, users can screen reviews they have

already uploaded from the web app. The most notable fea-

ture is the ability to use the app while offline. Users first

download the entire review while online then work on it

even in the absence of a network connection, and then,

once connected, the app will automatically sync back to

the Rayyan servers.

Predicting included and excluded studies

An important feature of the Rayyan app is its ability to

learn from users’ decisions to include or exclude studies

which can then be used to build a model that would allow

Ouzzani et al. Systematic Reviews

(2016) 5:210

Page 5 of 10

Fig. 2 Rayyan dashboard. The dashboard lists all reviews for this user as well as for each review the progress in terms of decisions made and

estimated time spent working on the review for all collaborators

suggestions to be offered on studies that are awaiting

screening. More specifically, after removing stop words

and stemming the remaining words from the title and

abstract, Rayyan extracts all the words (unigrams) and

pairs of words (bigrams) and previously computed MeSH

terms. These are then used as features by a support vec-

tor machine (SVM) classifier [15]. As users label citations

to studies as excluded or included, Rayyan calls the SVM

classifier which learns the features of these excluded and

included citations and builds a model, or classifier, accord-

ingly. The classifier then runs on the citations that await

labeling and outputs a score of how close each study

matches the include and exclude classes. That score is

then turned into a five-star rating that is presented to

the user. As the user continues to label more citations, if

Rayyan believes it can improve its prediction quality, then

itwillusethesenewlabeledexamplestoproduceanew

model and then run it on the remaining non-labeled cita-

tions. This process is repeated until there are no more

citations to label or the model cannot be improved any

further.

Results and Discussions

Evaluating the prediction algorithm

To test the quality of Rayyan’s SVM classifier, we used the

abovefeaturesonacollectionofsystematicreviewsfrom

a study published in [16]. In this study, test collections

were built for each of 15 review topics (Table 1) which had

been conducted by the Oregon EPC, Southern California

EPC, and Research Triangle Institute/University of North

Carolina (RTI/UNC) EPC. For each review, we know all

the articles and what was included/excluded. The ratio of

included articles ranged from 0.5 to 21.7%, with the largest

review containing 3465 studies and the smallest 310.

A twofold cross-validation was used with 50% of the

data going to training and 50% to testing. This process

was repeated ten times, and the results were averaged.

Two metrics were used for the evaluation of the quality

of the classifier, AUC and WSS@95. The ROC (receiver

operating characteristic) curve is obtained by graphing

the true positive rate against the false positive rate as we

vary the threshold used by the classifier. AUC refers sim-

ply to the area under this curve; 1.0 is a perfect score and

0.5 is equivalent to a random ordering. The work saved

over random sampling measured at 0.95 recall (WSS@95),

introduced in [16], refers to the percentage of studies that

the reviewers do not have to go through because they have

been screened out by the classifier at a recall of 0.95, com-

pared to random sampling. WSS =

TN+FN

N

− (1 − Recall)

where TN is the number of true negatives, FN is the

number of false negatives, and N is the total number of

instances in the dataset. Recall refers to the recall of the

positive class (included studies). The results we obtained

are AUC = 0.87 ± 0.09 and WSS@95 = 0.49 ± 0.18.

The 49% result is important since it shows that Rayyan

can help save time using the automatic prediction. While

Rayyan-a web and mobile app for systematic reviews.

Citations

References

"Rayyan-a web and mobile app for sys..." refers background in this paper

"Rayyan-a web and mobile app for sys..." refers background in this paper

Related Papers (5)

Trending Questions (1)