Sample size considerations for the external validation of a multivariable prognostic model: a resampling study.
Reads0
Chats0
TLDR
This study suggests that externally validating a prognostic model requires a minimum of 100 events and ideally 200 (or more) events, and provides guidance on sample size for investigators designing an external validation study.Abstract:
After developing a prognostic model, it is essential to evaluate the performance of the model in samples independent from those used to develop the model, which is often referred to as external validation. However, despite its importance, very little is known about the sample size requirements for conducting an external validation. Using a large real data set and resampling methods, we investigate the impact of sample size on the performance of six published prognostic models. Focussing on unbiased and precise estimation of performance measures (e.g. the c-index, D statistic and calibration), we provide guidance on sample size for investigators designing an external validation study. Our study suggests that externally validating a prognostic model requires a minimum of 100 events and ideally 200 (or more) events.read more
Citations
More filters
Journal ArticleDOI
Prediction models for diagnosis and prognosis of covid-19: systematic review and critical appraisal
Laure Wynants,Laure Wynants,Ben Van Calster,Ben Van Calster,Gary S. Collins,Gary S. Collins,Richard D Riley,Georg Heinze,Ewoud Schuit,Marc J.M. Bonten,Darren Dahly,Johanna A A G Damen,Thomas P. A. Debray,Valentijn M.T. de Jong,Maarten De Vos,Paula Dhiman,Paula Dhiman,Maria C Haller,Michael O. Harhay,Liesbet Henckaerts,Pauline Heus,Michael Kammer,Nina Kreuzberger,Anna Lohmann,Kim Luijken,Jie Ma,Glen P. Martin,David J. McLernon,Constanza L Andaur Navarro,Johannes B. Reitsma,Jamie C. Sergeant,Chunhu Shi,Nicole Skoetz,Luc J.M. Smits,Kym I E Snell,Matthew Sperrin,René Spijker,René Spijker,Ewout W. Steyerberg,Toshihiko Takada,Ioanna Tzoulaki,Ioanna Tzoulaki,Sander M. J. van Kuijk,Bas C T van Bussel,Bas C T van Bussel,Iwan C. C. van der Horst,Florien S. van Royen,Jan Y Verbakel,Jan Y Verbakel,Christine Wallisch,Christine Wallisch,Jack Wilkinson,Robert Wolff,Lotty Hooft,Karel G.M. Moons,Maarten van Smeden +55 more
TL;DR: Proposed models for covid-19 are poorly reported, at high risk of bias, and their reported performance is probably optimistic, according to a review of published and preprint reports.
Journal ArticleDOI
Calculating the sample size required for developing a clinical prediction model.
Richard D Riley,Joie Ensor,Kym I E Snell,Frank E. Harrell,Glen P. Martin,Johannes B. Reitsma,Karel G.M. Moons,Gary S. Collins,Maarten van Smeden,Maarten van Smeden,Maarten van Smeden +10 more
TL;DR: In this article, the authors provide guidance on how to calculate the sample size required to develop a clinical prediction model.
Journal ArticleDOI
PROBAST : A Tool to Assess Risk of Bias and Applicability of Prediction Model Studies: Explanation and Elaboration
Karel G.M. Moons,Robert Wolff,Richard D Riley,Penny Whiting,Marie Westwood,Gary S. Collins,Johannes B. Reitsma,Jos Kleijnen,Susan Mallett +8 more
TL;DR: The rationale behind the domains and signaling questions, how to use them, and how to reach domain-level and overall judgments about ROB and applicability of primary studies to a review question are described.
Journal ArticleDOI
A calibration hierarchy for risk models was defined: from utopia to empirical data.
Ben Van Calster,Ben Van Calster,Daan Nieboer,Yvonne Vergouwe,Bavo De Cock,Michael J. Pencina,Ewout W. Steyerberg +6 more
TL;DR: Strong calibration is desirable for individualized decision support but unrealistic and counter productive by stimulating the development of overly complex models, and model development and external validation should focus on moderate calibration.
Journal ArticleDOI
External validation of clinical prediction models using big datasets from e-health records or IPD meta-analysis: opportunities and challenges
Richard D Riley,Joie Ensor,Kym I E Snell,Thomas P. A. Debray,Doug G Altman,Karel G.M. Moons,Gary S. Collins +6 more
TL;DR: Novel opportunities for external validation in big, combined datasets in e-health records and individual participant data meta-analysis are illustrated, drawing attention to methodological challenges and reporting issues.
References
More filters
Book
An introduction to the bootstrap
Bradley Efron,Robert Tibshirani +1 more
TL;DR: This article presents bootstrap methods for estimation, using simple arguments, with Minitab macros for implementing these methods, as well as some examples of how these methods could be used for estimation purposes.
Journal ArticleDOI
Multivariable prognostic models: issues in developing models, evaluating assumptions and adequacy, and measuring and reducing errors
TL;DR: In this article, an easily interpretable index of predictive discrimination as well as methods for assessing calibration of predicted survival probabilities are discussed, which are particularly needed for binary, ordinal, and time-to-event outcomes.
BookDOI
Regression modeling strategies : with applications to linear models, logistic regression, and survival analysis
TL;DR: In this article, the authors present a case study in least squares fitting and interpretation of a linear model, where they use nonparametric transformations of X and Y to fit a linear regression model.
Journal ArticleDOI
General Cardiovascular Risk Profile for Use in Primary Care The Framingham Heart Study
Ralph B. D'Agostino,Ramachandran S. Vasan,Michael J. Pencina,Philip A. Wolf,Mark R. Cobain,Joseph M. Massaro,William B. Kannel +6 more
TL;DR: A sex-specific multivariable risk factor algorithm can be conveniently used to assess general CVD risk and risk of individual CVD events (coronary, cerebrovascular, and peripheral arterial disease and heart failure) and can be used to quantify risk and to guide preventive care.
Book ChapterDOI
Prognostic/Clinical Prediction Models: Multivariable Prognostic Models: Issues in Developing Models, Evaluating Assumptions and Adequacy, and Measuring and Reducing Errors
TL;DR: An easily interpretable index of predictive discrimination as well as methods for assessing calibration of predicted survival probabilities are discussed, applicable to all regression models, but are particularly needed for binary, ordinal, and time-to-event outcomes.