Page 1 of 29
PROBAST: A tool to assess the risk of bias and applicability
of prediction model studies
Robert F. Wolff
1,#
, Karel G. M. Moons
2,3,#
, Richard D. Riley
4
, Penny F. Whiting
5,6
, Marie Westwood
1
,
Gary S. Collins
7
, Johannes B. Reitsma
2,3
, Jos Kleijnen
1,8
, Susan Mallett
9
on behalf of the PROBAST
group
*
1
Kleijnen Systematic Reviews Ltd, York, United Kingdom
2
Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht
University, Utrecht, The Netherlands
3
Cochrane Netherlands, UMC Utrecht, Utrecht University, The Netherlands
4
Centre for Prognosis Research, Research Institute for Primary Care and Health Sciences, Keele
University, Keele, United Kingdom
5
Bristol Medical School, University of Bristol, Bristol, United Kingdom
6
NIIHR CLAHRC West, University Hospitals Bristol NHS Foundation Trust, Bristol, United Kingdom
7
Centre for Statistics in Medicine, Nuffield Department of Orthopaedics, Rheumatology and
Musculoskeletal Diseases, University of Oxford, Oxford, United Kingdom
8
School for Public Health and Primary Care (CAPHRI) Maastricht University, Maastricht, The
Netherlands
9
Institute of Applied Health Research, NIHR Birmingham Biomedical Research Centre, College of
Medical and Dental Sciences, University of Birmingham, Birmingham, United Kingdom
#
Both authors contributed equally
*
All members of the PROBAST group are listed in the appendix
Corresponding author:
Dr Robert Wolff
Kleijnen Systematic Reviews Ltd
Unit 6
Escrick Business Park
Riccall Road
Escrick
York YO19 6FD
United Kingdom
Tel. +44 (0)1904 727987
Fax. +44 (0)1904 720429
Email. robert@systematic-reviews.com
Short title: PROBAST
Word count: 3,008 words (Introduction; Methods; Results; Discussion)
Keywords: Bias (Epidemiology); Diagnosis, Evidence-Based Medicine; Multivariable Analysis;
Prediction; Prognosis; Reproducibility of Results
Page 2 of 29
Abstract
(309 words)
Background: Clinical prediction models combine several predictors (risk or prognostic factors) to
estimate the risk whether a particular condition is present (diagnostic model) or whether a certain
event will occur in the future (prognostic model). Large numbers of diagnostic and prognostic
prediction model studies are published each year and a tool facilitating their quality assessment is
needed, e.g. to support systematic reviews and evidence syntheses.
Objective: To introduce and describe the development of PROBAST, a tool for assessing the risk of
bias and applicability of prediction model studies.
Methods: Web-based Delphi procedure (involving 40 experts in the field of prediction model
research) and refinement of the tool through piloting. The scope of PROBAST was determined with
consideration of existing risk of bias tools and reporting guidelines, such as CHARMS, QUADAS,
QUIPS, and TRIPOD.
Results: After seven Delphi rounds, a final tool was developed which utilises a domain-based
structure supported by signalling questions. PROBAST assesses the risk of bias of prediction model
studies and any concerns for their applicability. Studies that PROBAST can be used for include those
developing, validating, and extending a prediction model. We define risk of bias to occur when
shortcomings in the study design, conduct or analysis lead to systematically distorted estimates of
model predictive performance or to an inadequate model to address the research question. The
predictive performance is typically evaluated using calibration and discrimination, and
sometimes (notably in diagnostic model studies) classification measures. Applicability refers to the
extent to which the prediction model study matches the systematic review question in terms of the
target population, predictors, or outcomes of interest.
PROBAST comprises 20 signalling questions grouped into four domains: participant selection,
predictors, outcome, and analysis.
Conclusions: PROBAST can be used to assess the risk of bias and any concerns for applicability of
studies developing, validating or extending (adjusting) prediction, both diagnostic and prognostic,
models.
Page 3 of 29
Introduction
(415 words)
Prediction relates to determining the probability of something currently unknown. In the context of
medical research, prediction typically relates to either diagnosis (probability of a certain condition
being present but not yet detected) or prognosis (probability of developing a future outcome).(1-3)
Prognosis does not only pertain to sick individuals or with an established diagnosis, but also to, for
example, prognosis of pregnant women at risk of developing diabetes(4) or of individuals in the
general population at risk of developing osteoporotic fractures(5).
Prediction research includes predictor finding studies, prediction model development, validation and
adjusting or updating studies, and prediction model impact studies.(1) Predictor finding studies (also
known as risk factor or prognostic factor studies) aim to identify which predictors independently
contribute to the prediction of a diagnostic or prognostic outcome.(1, 6) Prediction model studies
typically aim to develop, validate or adjust (e.g. extend) a multivariable prediction model. In a
prediction model, multiple predictors are used in combination for estimating individual probabilities
to inform and often guide individual care.(2, 7, 8) These models can either predict an individual’s
probability of currently having a particular outcome or disease (diagnostic prediction model) or
experiencing a particular outcome in the future (prognostic prediction model). Well known examples
are the Wells rule for diagnosing deep venous thrombosis,(9) the Ottawa ankle rules for detecting
fractures,(10, 11) QRISK2 for predicting cardiovascular risk,(12) and the PCPT risk calculator for
prostate biopsies.(13)
Prediction models, both diagnostic and prognostic, are widely used for a variety of medical domains
and settings,(14-16) evidenced by the large number of models developed, especially in cancer,(17,
18) neurology,(19, 20) and cardiovascular disease domains.(21) Prediction models are sometimes
described as risk prediction models, predictive models, prediction indices or rules, or risk scores.(2, 8)
Prediction model impact studies evaluate the effect of using a model to guide patient care compared
to not using such a model, and focus on the effect of its use on clinical decision making, patient
outcomes, or costs of care, using a comparative design.(1)
Systematic reviews have a key role in evidence based medicine and in the development of clinical
guidelines.(22-24) They are considered to provide the most reliable form of evidence for the effects
of an intervention or diagnostic test.(25, 26) Review and synthesis of prediction model studies is a
relatively new and evolving area. Systematic reviews of prediction models are increasingly
undertaken to appraise and summarise evidence on the performance of prediction models.(1, 7, 27)
They typically aim to systematically identify, appraise and summarise primary studies reporting the
development or validation of one or more prediction models.(27)
Quality assessment of included studies is a crucial step in any systematic review.(25, 26) The QUIPS
tool has been developed to assess the risk of bias in predictor finding (prognostic factor) studies.(28)
The methodological quality of studies investigating the impact of a prediction model using a
comparative randomised design can be assessed using the revised Cochrane risk of bias
tool (ROB 2.0)(29) or ROBINS-I for non-randomised comparative designs.(30) With the increased
numbers of systematic reviews for prediction model studies, a tool facilitating quality assessment for
individual prognostic and diagnostic prediction model studies is urgently needed.
Page 4 of 29
We present PROBAST (Prediction model Risk Of Bias ASsessment Tool), a tool to appraise the quality
of prediction model studies. The tool allows the assessment of risk of bias and concerns for the
applicability of diagnostic and prognostic prediction model studies. PROBAST can be used to assess
both model development and model validation studies, including those adjusting (e.g. extending) a
prediction model (Box 1). We explicitly refer to the accompanying Explanation and Elaboration (E&E)
paper for detailed explanations on how to use the PROBAST tool and how to make risk of bias and
applicability judgements.[REF E&E paper] To the best of our knowledge, PROBAST is the first tool
which has been rigorously developed for this purpose.
Page 5 of 29
Methods – Development of PROBAST
(840 words)
Development of PROBAST was based on a four-stage approach for developing health research
reporting guidelines: define the scope, review the evidence base, web-based Delphi procedure, and
refine the tool through piloting.(31) Guidelines explicitly aimed at the development of quality
assessment tools were not available at the time.(32)
Development stage 1: Define the scope
A steering group of nine experts in the area of prediction model studies and quality assessment tool
development was established. This group agreed on key features of the desired scope of PROBAST
through regular teleconferences and face-to-face meetings. The scope was further refined during the
web-based Delphi procedure with a panel of 40 experts.
It was agreed that PROBAST should not cover all multivariable diagnostic or prognostic studies but
only primary studies that developed, validated or adjusted (e.g. extended) one or more multivariable
prediction models for diagnosis or prognosis. A multivariable prediction model is defined as any
combination or equation of two or more predictors for estimating the probability or risk of a
diagnostic or prognostic outcome for an individual.(7, 8, 33-35) Hence, a relevant primary prediction
model study was one that included a model development, model validation or model adjustment (or
a combination of these) for the purpose of making individualised predictions of a diagnostic or
prognostic outcome (Box 1). Diagnostic and prognostic model studies often use different terms for
the predictors and outcomes (Box 2). Studies using multivariable modelling techniques to identify
predictors (e.g. risk or prognostic factors) associated with an outcome but not attempting to develop,
validate or adjust (e.g. extend) a model for making individualised predictions are not covered by
PROBAST.(6) Hence PROBAST is not intended for predictor finding studies and prediction model
impact studies.
PROBAST was designed to assess primary studies included in a systematic review. The group agreed
that PROBAST would assess both the risk of bias and concerns for applicability of a study that
evaluates (develops, validates or extends) a multivariable diagnostic or prognostic prediction model
to be used for individualised predictions. A domain-based structure was adopted similar to that used
in other risk of bias tools such as the revised Cochrane Risk of Bias tool,(29) QUADAS-2,(36) ROBINS-
I(30) and ROBIS.(37)
Development stage 2: Review the evidence
Three different approaches were used to provide an evidence base to inform the development of
PROBAST: (1) identification of relevant methodological reviews in the area of prediction model
research, (2) asking members of the steering group to identify relevant methodological studies, and
(3) use of the Delphi procedure to ask members of the wider group to identify additional evidence.
Identified literature was used to guide the scope and produce an initial list of signalling questions for
consideration for inclusion in PROBAST.(1, 2, 6-8, 34, 35, 38-44) Signalling questions were grouped
into common themes in order to identify possible domains.
