Transparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis (TRIPOD): explanation and elaboration.

TL;DR: In virtually all medical domains, diagnostic and prognostic multivariable prediction models are being developed, validated, updated, and implemented with the aim to assist doctors and individuals in estimating probabilities and potentially influence their decision making.

Abstract: The TRIPOD (Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis) Statement includes a 22-item checklist, which aims to improve the reporting of studies developing, validating, or updating a prediction model, whether for diagnostic or prognostic purposes. The TRIPOD Statement aims to improve the transparency of the reporting of a prediction model study regardless of the study methods used. This explanation and elaboration document describes the rationale; clarifies the meaning of each item; and discusses why transparent reporting is important, with a view to assessing risk of bias and clinical usefulness of the prediction model. Each checklist item of the TRIPOD Statement is explained in detail and accompanied by published examples of good reporting. The document also provides a valuable reference of issues to consider when designing, conducting, and analyzing prediction model studies. To aid the editorial process and help peer reviewers and, ultimately, readers and systematic reviewers of prediction model studies, it is recommended that authors include a completed checklist in their submission. The TRIPOD checklist can also be downloaded from www.tripod-statement.org.

Summary

DIAGNOSTIC AND PROGNOSTIC PREDICTION MODELS

• Multivariable prediction models fall into 2 broad categories: diagnostic and prognostic prediction models (Box A).
• They are developed from and to be used for individuals suspected of having that condition.
• They may be models for either ill or healthy individuals.
• Prognostic models include models to predict recurrence, complications, or death in a certain period after being diagnosed with a particular disease.
• The authors refer to both diagnostic and prognostic prediction models as “prediction models,” highlighting issues that are specific to either type of model.

OF PREDICTION MODELS

• Prediction model studies may address the development of a new prediction model (10), a model evaluation (often referred to as model validation) with or without updating of the model [19–21]), or a combination of these .
• In other words, in diagnostic prediction the interest is in principle a cross-sectional relationship, whereas prognostic prediction involves a longitudinal relationship.
• Nevertheless, in diagnostic modeling studies, for logistical reasons, a time window between predictor (index test) measurement and the reference standard is often necessary.
• Studies developing new prediction models should therefore always include some form of internal validation to quantify any optimism in the predictive performance (for example, calibration and discrimination) of the developed model and adjust the model for overfitting.
• With a single data set, temporal splitting and model validation can be considered intermediate between internal and external validation.

INCOMPLETE AND INACCURATE REPORTING

• Prediction models are becoming increasingly abundant in the medical literature (9, 33, 34), and policymakers are increasingly recommending their use Box B. Similarities and differences between diagnostic and prognostic prediction models.
• Different terms for similar features between diagnostic and prognostic modeling studies are summarized below.
• Given the abundance of published prediction models across almost all clinical domains, critical appraisal and synthesis of the available reports is a requirement to enable readers, care providers, and policymakers to judge which models are useful in which situations.
• Prediction model development studies with validation* in other participant data have the same aims as the previous type, but the development of the model is followed by quantifying the model's predictive performance in participant data other than the development data set .
• The aim of model validation is to evaluate the model's predictive performance in either resampled participant data of the development data set (often referred to as internal validation) or in other, independent participant data that were not used for developing the model (often referred to as external validation).

THE TRIPOD STATEMENT

• Prediction model studies can be subdivided into 5 broad categories (1, 8–10, 19, 20, 28, 33, 102–104): 1) prognostic or diagnostic predictor finding studies, 2) prediction model development studies without external validation, 3) prediction model development studies with external validation, 4) prediction model validation studies, and 5) model impact studies.
• TRIPOD addresses the reporting of prediction model studies aimed at developing or validating 1 or more prediction models (Box C).
• These development and validation studies can in turn be subdivided into various types .
• Prognostic or diagnostic predictor finding studies and model impact studies often have different aims, designs, and reporting issues compared with studies developing or validating prediction models.
• Prediction model studies aim to quantify the effect of using a prediction model on participant and physician decision making or directly on participant health outcomes, relative to not using the model (20, 102, 107).

DEVELOPMENT OF TRIPOD

• The authors followed published guidance for developing reporting guidelines (113) and established a steering committee (Drs. Collins, Altman, Moons, and Reitsma) to organize and coordinate the development of TRIPOD.
• Respondents (24 of 27) included methodologists, health care professionals, and journal editors.
• At the meeting, the results of the survey were presented, and each of the 76 candidate checklist items was discussed in turn.
• For each item, consensus was reached on whether to retain, merge with another item, or omit the item.

Aim and Outline of This Document

• The TRIPOD Statement is a checklist of 22 items considered essential for good reporting of studies developing or validating multivariable prediction models (Table 1) (114).
• Describe the study design or source of data (e.g., randomized trial, cohort, or registry data), separately for the development and validation data sets, if applicable 4b D;V Specify the key study dates, including start of accrual; end of accrual; and, if applicable, end of follow-up Participants 5a D;V.
• The authors have split the discussion of a few complex and lengthy items into multiple parts to aid clarity.

Use of Examples

• For each item, the authors present examples from published articles of both development and validation of prediction models, and often for both diagnosis and prognosis; they illustrate the type of information that is appropriate to report.
• The authors use of a particular example does not imply that all aspects of the study were well conducted and reported, or that the methods being reported are necessarily the best methods to be used in prediction model research.
• Rather, the examples illustrate a particular aspect of an item that has been well reported in the context of the methods used by the study authors.
• Some of the quoted examples have been edited, with text omitted (denoted by . . . ), text added (denoted by [ ]), citations removed, or abbreviations spelled out, and some tables have been simplified.

USE OF TRIPOD

• Depending on the type of prediction model study (development, validation, or both), each checklist item (relevant to the study type) should be addressed somewhere in the report.
• If a particular checklist item cannot be addressed, acknowledgment that the information is unknown or irrelevant (if so) should be clearly reported.
• Authors may find it convenient to report information for some of the requested items in a supplementary material section (for example, in online appendices).
• Announcements and information relating to TRIPOD will be broadcast on the TRIPOD Twitter address (@TRIPODStatement).

Title

• Identify the study as developing and/or validating a multivariable prediction model, the target population, and the outcome to be predicted. [D;V].
• Give an overall interpretation of the results, considering objectives, limitations, results from similar studies, and other relevant evidence Implications 20 D;V.
• The authors recommend using the TRIPOD Checklist in conjunction with the TRIPOD Explanation and Elaboration document.
• Development and external validation of prognostic model for 2 year survival of non small cell lung cancer patients treated with chemoradiotherapy (116).
• Predicting the 10 year risk of cardiovascular disease in the United Kingdom: independent and external validation of an updated version of QRISK2 (117).

Examples of Well-Known Models

• Validation of the Framingham coronary heart disease prediction scores: results of a multiple ethnic groups investigation (120).
• External validation of the SAPS II APACHE II and APACHE III prognostic models in South England: a multicentre study (121).

Explanation

• Studies of prediction, even prospective studies, tend to receive little or no funding, which has been suggested to contribute to the large number of poorquality studies: many are conducted without any peer review during the planning phase, when funding is usually sought (472).
• Authors should disclose all sources of funding received for conducting the study and state what role of the funder had in the design, conduct, analysis, and reporting of the study.
• If the funder had no involvement, the authors should state so.
• Similarly, if the study received no external funding, the authors should clearly say so.
• For models that are incorporated in guidelines, it is important to show the potential financial and other conflicts of interest of all guideline development members, not just those involved in the prediction model development (316, 483, 484).

Examples

• The Reynolds Risk Score Project was supported by investigator-initiated research grants from the Donald W. Reynolds Foundation (Las Vegas, Nev) with additional support from the Doris Duke Charitable Foundation (New York, NY), and the Leducq Foundation (Paris, France).
• The Women's Health Study cohort is supported by grants from the National Heart, Lung, and Blood Institute and the National Cancer Institute (Bethesda, Md) (208).
• The Clinical and Translational Service Center at Weill Cornell Medical College provided partial support for data analyses.
• The funding source had no role in the design of their analyses, its interpretation, or the decision to submit the manuscript for publication (380).

Source of Data

• Describe the study design or source of data (for example, randomized trial, cohort, or registry data), separately for the development and validation data sets, if applicable. [D;V].
• For efficiency, in some diagnostic modeling studies the reference standard is performed first, and the study uses all the cases (patients with the target condition) but a random sample of the noncases.
• There may be concerns about the generalizability of a model developed or validated by using data from a randomized trial, owing to extensive exclusion criteria (1).
• The following year between 1 May and 30 June 2009 the authors recruited the validation cohort on the same basis: 302 consecutive patients aged 18 or over of both sexes who were admitted to any of the same four internal medicine wards at the hospital (179).
• Follow-up from inclusion may be the same for all participants, in which case the study duration should be specified.

Example

• Clinically meaningful interactions were included in the model.
• All interaction terms were removed as a group, and the model was refit if results were nonsignificant.
• Specifically, interactions between home use of -blockers and diuretics and between edema on physical examination and a history of heart failure were tested (284).

Blinding to Other Predictor Information

• Assessors of predictors requiring interpretation may also be provided with other information (for example, prior information obtained during assessment of medical history or physical examination).
• Unlike blinding for the outcome information when assessing predictors, blinding for information of other predictors is not per se good or bad.
• The appropriateness depends on the research question and the potential clinical application of the specific predictors (209, 225, 226).
• Interpretation of subsequent predictors with knowledge of prior predictor information may be specifically designed, if in daily practice these subsequent predictors are always interpreted in view of this prior information.
• If a research purpose is to quantify whether a particular predictor or test may replace another predictor or test (for example, whether positron emission tomography–computed tomography may replace traditional scanning for the detection of cancer lesions in the lungs), mutually blinding the observers of both for each others' results is indicated to prevent contamination in both interpretations (225, 239).

Development Study

• As discussed under item 10b, a model's performance is likely to be overestimated when it is developed and assessed for its predictive accuracy on the same data set (23).
• It is better to have a large sample in the first place.
• These concerns apply even when no predictor selection will be performed.
• They are far greater, however, when the predictors in the model will be selected from a large number of available predictors , especially when there are no strong predictors.

Examples: Flow of Participants

• Follow-up Time Median follow-up was computed according to the “reverse Kaplan Meier” method, which calculates potential follow-up in the same way as the Kaplan–Meier estimate of the survival function, but with the meaning of the status indicator reversed, also known as Examples.
• Such information is vital to judge the context in which the prediction model can be validated or applied.
• It can be helpful also to include other information in the diagram, such as numbers of participants with missing observations and the numbers of outcome events.
• Here, the standard Kaplan–Meier method is used with the event indicator reversed, so that censoring becomes the outcome of interest (108).
• For diagnostic studies with delayed disease verification as outcome (items 4a and 6b), reporting the median follow-up is important.

Model Development

• Specify the number of participants and outcome events in each analysis. [D].
• Therefore, if univariable associations are reported, the number of participants without missing values for each predictor and the corresponding number of events in those participants should be presented.
• Similarly, authors may derive or compare the performance of more than 1 multivariable model on the same data set.
• From reference 396. * Defined by any positive sputum or bronchoalveolar lavage mycobacterial culture on solid media.
• All but 1 patient had been taking cotrimoxazole for ≥1 month.

Model Specification

• Present the full prediction model to allow predictions for individuals (i.e., all regression coefficients, and model intercept or baseline survival at a given time point). [D].
• * -Coefficients for the variables included in the simplified (model A) and complete (model B) models fitted in the derivation cohort for myocardial infarction or angina, and models' performance in the validation cohort by sex.

Model Performance

• Report performance measures (with confidence intervals) for the prediction model. [D;V].
• If multiple models were developed or evaluated, then the performance measures for each model should be reported.
• Decision analytic measures, such as net benefit or relative utility, are usually presented graphically rather than as a single numerical estimate (361–363, 422).
• The range of the x-axis should generally be chosen to represent reasonable variation in practice.
• If 2 models being compared at a particular threshold have a difference in net benefit of 0.005 (that is, model A [QRISK22011] minus model B [NICE Framingham]), then this is interpreted as the net increase in true-positive findings—that is, by using model A, 5 more true-positive outcomes are identified per 1000 individuals without increasing the number of false-positive findings.

Limitations

• Discuss any limitations of the study (such as nonrepresentative sample, few events per predictor, missing data). [D;V].
• As more institutions incorporate electronic medical records into their process flow, models such as the one described here can be of great value.
• These differences in ICU stay are also likely to adversely impact the use of ICU day 5 as a threshold for concern about a prolonged stay.
• At the traditional threshold of 20% used to designate an individual at high risk of developing cardiovascular disease, the net benefit of QRISK2-2011 for men is that the model identified five more cases per 1000 without increasing the number treated unnecessarily when compared with the NICE Framingham equation.
• There is no gold standard for clinical assessment, and the best method of assessing clinical performance remains controversial.

Interpretation

• For validation, discuss the results with reference to performance in the development data, and any other validation data. [V].
• The ABCD2 score was a combined effort by teams led by Johnston and Rothwell, who merged two separate datasets to derive highrisk clinical findings for subsequent stroke.
• Rothwell's dataset was small, was derived from patients who had been referred by primary care physicians and used predictor variables W54 Annals of Internal Medicine Vol. 162 No. 1 6 January 2015 www.annals.org Downloaded From: http://annals.org/ by a University Library Utrecht User on 12/03/2015 scored by a neurologist one to three days later.
• Tsivgoulis and coworkers supported using an ABCD2 score of more than 2 as the cutoff for high risk based on the results of a small prospective study of patients who had a transient ischemic attack and were admitted to hospital.

CONCLUDING REMARKS

• Studies addressing prediction models are abundant, with the number of publications describing the development, validation, updating, or extension of prediction models showing no sign of abating.
• The items included in the checklist reflect numerous discussions to reach consensus on the minimal set of information to report to enable an informed assessment of study quality, risks of bias and clinical relevance, and enable the results to be used (532).
• The TRIPOD Web site (www.tripod-statement.org) will provide a forum for discussion, suggestions for improving the checklist and this explanation and elaboration document, and resources relevant to prediction model studies.
• Karel G.M. Moons, PhD, Julius Centre for Health Sciences and Primary Care, UMC Utrecht, PO Box 85500, 3508 GA Utrecht, the Netherlands; e-mail, K.G.M.Moons@umcutrecht.nl, also known as Requests for Single Reprints.

1. Moons KG, Royston P, Vergouwe Y, Grobbee DE, Altman DG.

• A Practical Approach to Development, Validation, and Updating, also known as Clinical Prediction Models.
• Jacob M, Lewsey JD, Sharpin C, Gimson A, Rela M, van der Meulen JH.
• Evaluating diagnostic accuracy in the face of multiple reference standards.

Figures

Table 14. Example Table: Presenting Both the Original and Updated Prediction Model

Table 7. Example Table: Comparison of Participant Characteristics in Development and Validation Data [Development; Validation]

Table 18. Example of a Reclassification Table (With Net Reclassification Improvement and 95% CI) for a Basic and Extended Diagnostic Model Using a Single Probability

Figure 10. Example figure: a decision curve analysis.

Figure 2. Selection of predictors in a study of the development of a multivariable prediction model.

Table 6. Example Table: Participant Characteristics

Table 13. Example Table: Presenting the Full Diagnostic (Logistic) Model, Including the Intercept*

Table 11. Example Table: Unadjusted Association Between Each Predictor and Outcome*

Table 12. Example Table: Presenting the Full Prognostic (Survival) Model, Including the Baseline Survival, for a

Table 10. Example Table: Reporting the Number of Events in Each Unadjusted Analysis

Figure 6. Example figure: a graphical scoring system to obtain a predicted probability in an individual.

Figure 4. Example figure: participant flow diagram.

Figure 7. Example figure: a nomogram, and how to use it to obtain a predicted probability in an individual.

Table 3. Overview of Different Approaches for Updating an Existing Prediction Model*

Table 8. Example Table: Comparison of Participant Characteristics in Development and Validation Data [Validation]

Table 5. Example Table: Participant Characteristics

Figure 8. Example figure: a calibration plot with c-statistic and distribution of the predicted probabilities for individuals with and without the outcome (coronary

Figure 9. Example figure: a receiver-operating characteristic curve, with predicted risks labelled on the curve.

Figure 1. Types of prediction model studies covered by the TRIPOD statement.

Table 16. Example Table: A Simple Scoring System From Which Individual Outcome Risks (Probabilities) Can Be

Table 2. Example Table: Reporting Key Study Characteristics [Diagnosis; Development; Validation]

Full text

Transparent Reporting of a multivariable prediction model for
Individual Prognosis Or Diagnosis (TRIPOD): Explanation and
Elaboration
Karel G.M. Moons, PhD; Douglas G. Altman, DSc; Johannes B. Reitsma, MD, PhD; John P.A. Ioannidis, MD, DSc;
Petra Macaskill, PhD; Ewout W. Steyerberg, PhD; Andrew J. Vickers, PhD; David F. Ransohoff, MD; and Gary S. Collins, PhD
The TRIPOD (Transparent Reporting of a multivariable prediction
model for Individual Prognosis Or Diagnosis) Statement includes
a 22-item checklist, which aims to improve the reporting of stud-
ies developing, validating, or updating a prediction model,
whether for diagnostic or prognostic purposes. The TRIPOD
Statement aims to improve the transparency of the reporting of a
prediction model study regardless of the study methods used.
This explanation and elaboration document describes the ratio-
nale; clarifies the meaning of each item; and discusses why trans-
parent reporting is important, with a view to assessing risk of bias
and clinical usefulness of the prediction model. Each checklist
item of the TRIPOD Statement is explained in detail and accom-
panied by published examples of good reporting. The docu-
ment also provides a valuable reference of issues to consider
when designing, conducting, and analyzing prediction model
studies. To aid the editorial process and help peer reviewers
and, ultimately, readers and systematic reviewers of prediction
model studies, it is recommended that authors include a com-
pleted checklist in their submission. The TRIPOD checklist can
also be downloaded from www.tripod-statement.org.
Ann Intern Med. 2015;162:W1-W73. doi:10.7326/M14-0698 www.annals.org
For author affiliations, see end of text.
For members of the TRIPOD Group, see the Appendix.
I
n medicine, numerous decisions are made by care
providers, often in shared decision making, on the
basis of an estimated probability that a specific disease
or condition is present (diagnostic setting) or a specific
event will occur in the future (prognostic setting) in an
individual. In the diagnostic setting, the probability that
a particular disease is present can be used, for exam-
ple, to inform the referral of patients for further testing,
to initiate treatment directly, or to reassure patients that
a serious cause for their symptoms is unlikely. In the
prognostic context, predictions can be used for plan-
ning lifestyle or therapeutic decisions on the basis of
the risk for developing a particular outcome or state of
health within a specific period (1–3). Such estimates
of risk can also be used to risk-stratify participants in
therapeutic intervention trials (4–7).
In both the diagnostic and prognostic setting,
probability estimates are commonly based on combin-
ing information from multiple predictors observed or
measured from an individual (1, 2, 8–10). Information
from a single predictor is often insufficient to provide
reliable estimates of diagnostic or prognostic probabil-
ities or risks (8, 11). In virtually all medical domains,
diagnostic and prognostic multivariable (risk) predic-
tion models are being developed, validated, updated,
and implemented with the aim to assist doctors and
individuals in estimating probabilities and potentially
influence their decision making.
A multivariable prediction model is a mathematical
equation that relates multiple predictors for a particular
individual to the probability of or risk for the presence
(diagnosis) or future occurrence (prognosis) of a partic-
ular outcome (10, 12). Other names for a prediction
model include risk prediction model, predictive model,
prognostic (or prediction) index or rule, and risk score
(9).
Predictors are also referred to as covariates, risk
indicators, prognostic factors, determinants, test results,
or—more statistically—independent variables. They may
range from demographic characteristics (for example,
age and sex), medical history–taking, and physical ex-
amination results to results from imaging, electrophys-
iology, blood and urine measurements, pathologic ex-
aminations, and disease stages or characteristics, or
results from genomics, proteomics, transcriptomics,
pharmacogenomics, metabolomics, and other new bi-
ological measurement platforms that continuously
emerge.
DIAGNOSTIC AND PROGNOSTIC PREDICTION
MODELS
Multivariable prediction models fall into 2 broad
categories: diagnostic and prognostic prediction mod-
els (Box A). In a diagnostic model, multiple—that is, 2 or
more—predictors (often referred to as diagnostic test
results) are combined to estimate the probability that a
certain condition or disease is present (or absent) at the
moment of prediction (Box B). They are developed
from and to be used for individuals suspected of hav-
ing that condition.
In a prognostic model, multiple predictors are
combined to estimate the probability of a particular
outcome or event (for example, mortality, disease re-
currence, complication, or therapy response) occurring
in a certain period in the future. This period may range
from hours (for example, predicting postoperative
See also:
Related article .............................55
Editorial comment ..........................73
Annals of Internal Medicine RESEARCH AND REPORTING METHODS
© 2015 American College of Physicians W1
Downloaded From: http://annals.org/ by a University Library Utrecht User on 12/03/2015

complications [13]) to weeks or months (for example,
predicting 30-day mortality after cardiac surgery [14])
or years (for example, predicting the 5-year risk for de-
veloping type 2 diabetes [15]).
Prognostic models are developed and are to be
used in individuals at risk for developing that outcome.
They may be models for either ill or healthy individuals.
For example, prognostic models include models to
predict recurrence, complications, or death in a certain
period after being diagnosed with a particular disease.
But they may also include models for predicting the
occurrence of an outcome in a certain period in individ-
uals without a specific disease: for example, models to
predict the risk for developing type 2 diabetes (16) or
cardiovascular events in middle-aged nondiseased in-
dividuals (17), or the risk for preeclampsia in pregnant
women (18). We thus use prognostic in the broad
sense, referring to the prediction of an outcome in the
future in individuals at risk for that outcome, rather than
the narrower definition of predicting the course of pa-
tients who have a particular disease with or without
treatment (1).
The main difference between a diagnostic and
prognostic prediction model is the concept of time. Di-
agnostic modeling studies are usually cross-sectional,
whereas prognostic modeling studies are usually longi-
tudinal. In this document, we refer to both diagnostic
and prognostic prediction models as “prediction mod-
els,” highlighting issues that are specific to either type
of model.
DEVELOPMENT,VALIDATION, AND UPDATING
OF
PREDICTION MODELS
Prediction model studies may address the devel-
opment of a new prediction model (10), a model eval-
uation (often referred to as model validation) with or
without updating of the model [19 –21]), or a combina-
tion of these (Box C and Figure 1).
Model development studies aim to derive a predic-
tion model by selecting predictors and combining
them into a multivariable model. Logistic regression is
commonly used for cross-sectional (diagnostic) and
short-term (for example 30-day mortality) prognostic
outcomes and Cox regression for long-term (for exam-
ple, 10-year risk) prognostic outcomes. Studies may
also focus on quantifying the incremental or added
Box A.
Schematic representation of diagnostic and prognostic prediction modeling studies.
Predictors:
Patient characteristics
(symptoms & signs)
Imaging tests
Laboratory tests
Others
Diagnostic multivariable modeling study
Subjects with presenting
symptoms
Outcome:
Disease present
or absent
Outcome:
Development
of event Y
Cross-sectional
relationship
Predictors:
Patient characteristics
Disease characteristics
Imaging tests
Laboratory tests
Others
Prognostic multivariable modeling study
Subjects in a
health state
T = 0
T = 0
Longitudinal
relationship
End of
follow-u
p
Y Y Y
The nature of the prediction in diagnosis is estimating the probability that a specific outcome or disease is present (or absent) within an individual,
at this point in time—that is, the moment of prediction (T = 0). In prognosis, the prediction is about whether an individual will experience a specific
event or outcome within a certain time period. In other words, in diagnostic prediction the interest is in principle a cross-sectional relationship,
whereas prognostic prediction involves a longitudinal relationship. Nevertheless, in diagnostic modeling studies, for logistical reasons, a time
window between predictor (index test) measurement and the reference standard is often necessary. Ideally, this interval should be as short as
possible without starting any treatment within this period.
RESEARCH AND REPORTING METHODS The TRIPOD Statement: Explanation and Elaboration
W2 Annals of Internal Medicine • Vol. 162 No. 1 • 6 January 2015 www.annals.org
Downloaded From: http://annals.org/ by a University Library Utrecht User on 12/03/2015

predictive value of a specific predictor (for example,
newly discovered) (22) to a prediction model.
Quantifying the predictive ability of a model on the
same data from which the model was developed (often
referred to as apparent performance [Figure 1]) will
tend to give an optimistic estimate of performance, ow-
ing to overfitting (too few outcome events relative to
the number of candidate predictors) and the use of
predictor selection strategies (23–25). Studies develop-
ing new prediction models should therefore always in-
clude some form of internal validation to quantify any
optimism in the predictive performance (for example,
calibration and discrimination) of the developed model
and adjust the model for overfitting. Internal validation
techniques use only the original study sample and
include such methods as bootstrapping or cross-
validation. Internal validation is a necessary part of
model development (2).
After developing a prediction model, it is strongly
recommended to evaluate the performance of the
model in other participant data than was used for the
model development. External validation (Box C and
Figure 1) (20, 26) requires that for each individual in the
new participant data set, outcome predictions are
made using the original model (that is, the published
model or regression formula) and compared with the
observed outcomes. External validation may use partic-
ipant data collected by the same investigators, typically
using the same predictor and outcome definitions and
measurements, but sampled from a later period (tem-
poral or narrow validation); by other investigators in
another hospital or country (though disappointingly
rare [27]), sometimes using different definitions and
measurements (geographic or broad validation); in
similar participants, but from an intentionally different
setting (for example, a model developed in secondary
care and assessed in similar participants, but selected
from primary care); or even in other types of partici-
pants (for example, model developed in adults and as-
sessed in children, or developed for predicting fatal
events and assessed for predicting nonfatal events) (19,
20, 26, 28 –30). In case of poor performance (for exam-
ple, systematic miscalibration), when evaluated in an
external validation data set, the model can be updated
or adjusted (for example, recalibrating or adding a new
predictor) on the basis of the validation data set (Box C)
(2, 20, 21, 31).
Randomly splitting a single data set into model de-
velopment and model validation data sets is frequently
done to develop and validate a prediction model; this
is often, yet erroneously, believed to be a form of ex-
ternal validation. However, this approach is a weak and
inefficient form of internal validation, because not all
available data are used to develop the model (23, 32).
If the available development data set is sufficiently
large, splitting by time and developing a model using
data from one period and evaluating its performance
using the data from the other period (temporal valida-
tion) is a stronger approach. With a single data set,
temporal splitting and model validation can be consid-
ered intermediate between internal and external
validation.
INCOMPLETE AND INACCURATE REPORTING
Prediction models are becoming increasingly
abundant in the medical literature (9, 33, 34), and
policymakers are increasingly recommending their use
Box B.
Similarities and differences between diagnostic and prognostic prediction models.
Despite the different nature (timing) of the prediction, there are many similarities between diagnostic and prognostic prediction models, including:
• Type of outcome is often binary: either disease of interest present versus absent (in diagnosis) or the future occurrence of an event yes or no (in
prognosis).
• The key interest is to generate the probability of the outcome being present or occurring for an individual, given the values of 2 or more predictors, with
the purpose of informing patients and guiding clinical decision making.
• The same challenges as when developing a multivariable prediction model, such as selection of the predictors, model-building strategies, and handling of
continuous predictors and the danger of overfitting.
• The same measures for assessing model performance.
Different terms for similar features between diagnostic and prognostic modeling studies are summarized below.
Diagnostic Prediction Modeling Study
Diagnostic tests or index tests
Target disease/disorder (presence vs. absence)
Reference standard and disease verification
Partial verification
Explanatory variables, predictors, covariates (X variables)
Outcome (Y variable)
Missing outcomes
Prognostic Prediction Modeling Study
Prognostic factors or indicators
Event (future occurrence: yes or no)
Event definition and event measurement
Loss to follow-up and censoring
The TRIPOD Statement: Explanation and Elaboration RESEARCH AND REPORTING METHODS
www.annals.org Annals of Internal Medicine • Vol. 162 No. 1 • 6 January 2015 W3
Downloaded From: http://annals.org/ by a University Library Utrecht User on 12/03/2015

in clinical practice guidelines (35–40). For some specific
diseases, there is an overwhelming number of compet-
ing prediction models for the same outcome or target
population. For example, there are over 100 prognostic
models for predicting outcome after brain trauma (41),
over 100 models for prostate cancer (42), over 60 mod-
els for breast cancer prognosis (43), 45 models for car-
diovascular events after being diagnosed with diabetes
(44), over 40 models for predicting prevalent and inci-
dent type 2 diabetes (45), and 20 models for predicting
prolonged intensive care unit (ICU) stay after cardiac
surgery (46).
Given the abundance of published prediction
models across almost all clinical domains, critical ap-
praisal and synthesis of the available reports is a re-
quirement to enable readers, care providers, and
policymakers to judge which models are useful in
which situations. Such an assessment, in turn, is possi-
ble only if key details of how prediction models were
developed and validated are clearly reported (47, 48).
Only then can generalizability and risk of bias of pub-
lished prediction models be adequately assessed (49,
50), and subsequent researchers can replicate on the
same data, if needed, the steps taken to obtain the
same results (51, 52). Many reviews have illustrated,
however, that the quality of published reports that de-
scribe the development or validation of prediction
models across many different disease areas and differ-
ent journals is poor (3, 34, 41, 43, 45, 46, 48, 53–95).
For example, in a review of newly developed prediction
models in the cancer literature (54, 55), reporting was
disappointingly poor, with insufficient information pro-
vided about all aspects of model development. The
same was found in a recent review of prediction mod-
els for prevalent or incident type 2 diabetes (45) and of
prediction models published in 6 high-impact general
medical journals (34).
Reporting guidelines for randomized trials
(CONSORT [96]), observational studies (STROBE [97]),
tumor marker studies (REMARK [98]), molecular epide-
miology (STROBE-ME [99]), diagnostic accuracy
(STARD [100]), and genetic risk prediction studies
(GRIPS [101]) contain items that are relevant to all types
of studies, including those developing or validating
prediction models. The 2 guidelines most closely re-
lated to prediction models are REMARK and GRIPS.
However, the focus of the REMARK checklist is primarily
on prognostic factors and not prediction models,
whereas the GRIPS statement is aimed at risk prediction
using genetic risk factors and the specific methodolog-
ical issues around handling large numbers of genetic
variants.
To address a broader range of studies, we devel-
oped the TRIPOD guideline: Transparent Reporting of
a multivariable prediction model for Individual Progno-
sis Or Diagnosis. TRIPOD explicitly covers the develop-
ment and validation of prediction models for both di-
agnosis and prognosis, for all medical domains and all
types of predictors. TRIPOD also places considerable
emphasis on model validation studies and the report-
ing requirements for such studies.
Box C.
Types of prediction model studies.
Prediction model development studies without validation* in other
participant data aim to develop 1 (or more) prognostic or diagnostic
prediction model(s) from the data set at hand: the development set. Such
studies commonly aim to identify the important predictors for the
outcome, assign the mutually adjusted weights per predictor in a
multivariable analysis, develop a prediction model to be used for
individualized predictions, and quantify the predictive performance (e.g.,
discrimination, calibration, classification) of that model in the
development set. Sometimes, the development may focus on quantifying
the incremental or added predictive value of a specific (e.g., newly
discovered) predictor. In development studies, overfitting may occur,
particularly in small development data sets. Hence, development studies
ideally include some form of resampling techniques, such as
bootstrapping, jack-knife, or cross-validation. These methods quantify
any optimism in the predictive performance of the developed model and
what performance might be expected in other participants from the
underlying source population from which the development sample
originated (see Figure 1). These resampling techniques are often referred
to as "internal validation of the model," because no data other than the
development set are used; everything is estimated "internally" with the
data set at hand. Internal validation is thus always part of model
development studies (see Figure 1 and Box F).
Prediction model development studies with validation* in other
participant data have the same aims as the previous type, but the
development of the model is followed by quantifying the model's
predictive performance in participant data other than the development
data set (see Figure 1). This may be done in participant data collected by
the same investigators, commonly using the same predictor and outcome
definitions and measurements, but sampled from a later time period
(so-called "temporal" or "narrow" validation); by other investigators in
another hospital or country, sometimes using different definitions and
measurements (geographic or broad validation); in similar participants,
but from an intentionally chosen different setting (e.g., model developed
in secondary care and tested in similar participants, but selected from
primary care); or even in other types of participants (e.g., model
developed in adults and tested in children, or developed for predicting
fatal events and tested for predicting nonfatal events). Randomly
splitting a single data set into a development and a validation data set is
often erroneously referred to as a form of external validation* of the
model. But this is an inefficient form of "internal" rather than "external"
validation, because the 2 data sets only differ by chance (see Figure 1).
Model validation* studies without or with model updating aim to
assess and compare the predictive performance of 1 (or more) existing
prediction models by using participant data that were not used to
develop the prediction model. When a model performs poorly, a
validation study can be followed by updating or adjusting the existing
model (e.g., recalibrating or extending the model by adding newly
discovered predictors). In theory, a study may address only the updating
of an existing model in a new data set, although this is unlikely and
undesirable without first doing a validation of the original model in the
new data set (see Figure 1).
* The term validation, although widely used, is misleading, because it
indicates that model validation studies lead to a "yes" (good validation)
or "no" (poor validation) answer on the model's performance. However,
the aim of model validation is to evaluate (quantify) the model's
predictive performance in either resampled participant data of the
development data set (often referred to as internal validation) or in
other, independent participant data that were not used for developing
the model (often referred to as external validation).
RESEARCH AND REPORTING METHODS The TRIPOD Statement: Explanation and Elaboration
W4 Annals of Internal Medicine • Vol. 162 No. 1 • 6 January 2015 www.annals.org
Downloaded From: http://annals.org/ by a University Library Utrecht User on 12/03/2015

THE TRIPOD STATEMENT
Prediction model studies can be subdivided into 5
broad categories (1, 8 –10, 19, 20, 28, 33, 102–104): 1)
prognostic or diagnostic predictor finding studies, 2)
prediction model development studies without exter-
nal validation, 3) prediction model development stud-
ies with external validation, 4) prediction model valida-
tion studies, and 5) model impact studies. TRIPOD
addresses the reporting of prediction model studies
aimed at developing or validating 1 or more prediction
models (Box C). These development and validation
studies can in turn be subdivided into various types
(Figure 1). An increasing number of studies are evalu-
ating the incremental value (103) of a specific predictor,
to assess whether an existing prediction model
may need to be updated or adjusted (22, 105, 106).
TRIPOD also addresses such studies (Box C and
Figure 1).
Prognostic or diagnostic predictor finding studies
and model impact studies often have different aims,
designs, and reporting issues compared with studies
developing or validating prediction models. The for-
Figure 1.
Types of prediction model studies covered by the TRIPOD statement.
Type 1a Development of a prediction model where predictive performance is then directly evaluated using exactly the same data (apparent performance).
Type 1b Development of a prediction model using the entire data set, but then using resampling (e.g., bootstrapping or cross-validation) techniques to
evaluate the performance and optimism of the developed model. Resampling techniques, generally referred to as “internal validation”, are
recommended as a prerequisite for prediction model development, particularly if data are limited (6, 14, 15).
Type 2a The data are randomly split into 2 groups: one to develop the prediction model, and one to evaluate its predictive performance. This design is
generally not recommended or better than type 1b, particularly in case of limited data, because it leads to lack of power during model development
and validation (14, 15, 16).
Type 2b The data are nonrandomly split (e.g., by location or time) into 2 groups: one to develop the prediction model and one to evaluate its predictive
performance. Type 2b is a stronger design for evaluating model performance than type 2a, because allows for nonrandom variation between the
2 data sets (6, 13, 17).
Type 3 Development of a prediction model using 1 data set and an evaluation of its performance on separate data (e.g., from a different study).
Type 4 The evaluation of the predictive performance of an existing (published) prediction model on separate data (13).
Types 3 and 4 are commonly referred to as “external validation studies.” Arguably type 2b is as well, although it may be considered an intermediary between
internal and external validation.
D
Type 4: Validation only
Type 3: Development and validation
using separate data
Type 2b: Nonrandom split-sample
development and validation
Type 2a: Random split-sample
development and validation
Analysis
Type
Description
D V
DV
V
Type 1b: Development and validation
using resampling
Type 1a: Development only
Only a single data set
is available: All data
are used to develop
the model
Only a single data set
is available: A portion
of the data are used to
develop the model
Only a single data set
is available: A separate
data set is available
for validation
D = development data; V = validation data.
The TRIPOD Statement: Explanation and Elaboration RESEARCH AND REPORTING METHODS
www.annals.org Annals of Internal Medicine • Vol. 162 No. 1 • 6 January 2015 W5
Downloaded From: http://annals.org/ by a University Library Utrecht User on 12/03/2015

Frequently asked questions

Q1. Why was the computed tomography based procedure not included in the prediction models?

Since its data were used for imputation the computed tomography based procedure was not included as a predictor in the prediction models. 

Q2. What is the way to investigate the functional relationship of continuous predictors?

Spline functions, in particular restricted cubic splines, are another approach to investigate the functional relationship of continuous predictors (112). 

Q3. How many potential candidate variables were considered for inclusion in the AMI prediction rules?

Forty three potential candidate variables in addition to age and gender were considered for inclusion in the AMI [acute myocardial infarction] mortality prediction rules. 

Q4. What measures of discrimination improvement were used to assess the magnitude of the increase in model performance when?

The authors used 3 complementary measures of discrimination improvement to assess the magnitude of the increase in model performance when individual biomarkers were added to GRACE: change in AUC (ΔAUC), integrated discrimination improvement (IDI), and continuous and categorical net reclassification improvement (NRI). 

Q5. What are the possible considerations for exclusion of predictors before the actual statistical modeling?

Other possible considerations for predictor exclusion before the actual statistical modeling are that the predictor measurement was unreliable (58), or that relatively high financial costs or burden are associated with such measurement. 

Q6. What are the methods to visualize discrimination between persons without and with the outcome?

In addition to reporting a single numerical quantification, graphical approaches to visualize discrimination between persons without and with the outcome are recommended, such as a histogram, density plot, or dot plot for each outcome group (417). 

Q7. What was the way to achieve consistency across databases?

When definitions of variables were not identical across the different studies (for example physical activity), the authors tried to use the best available variables to achieve reasonable consistency across databases. 

Q8. what is the role of risk factors in the prediction of hip and osteoporotic fractures?

The use of clinical risk factors enhances the performance of BMD in the prediction of hip and osteoporotic fractures in men and women.