Evaluation of clinical prediction models: Decomposing performance uncertainty into model development and validation components
Abstract
There has been growing emphasis on acknowledging, quantifying, and communicating the uncertainty inherent in the output of clinical prediction models. However, allowing for model uncertainty when assessing the performance of an existing model in a new sample (i.e., external validation) has not been sufficiently explored. When validating a model that generates uncertain predictions, the uncertainty of a given performance metric is affected by both model uncertainty and the finiteness of the validation sample (validation uncertainty). For a given performance metric, we show that the total uncertainty of the performance estimate can be decomposed into the uncertainty due to the model and the uncertainty due to the validation. Also, we propose M2, the fraction of the total variance due to model uncertainty, and V2, the fraction of the total variance due to validation uncertainty. These measures serve as complementary scalar indices that can be implemented for common performance metrics and visualizations such as calibration plots using simple Monte Carlo algorithms. Critically, our case study shows that with the same model and the same validation sample, the contribution of uncertainty sources is highly variable across performance metrics (M2 varying from 12% for the Brier score to 78% for the calibration slope). This indicates that stakeholders who prioritize different aspects of performance (e.g., calibration versus clinical utility) might arrive at different decisions on whether future efforts should be directed towards creating a more reliable model or procuring a larger validation sample. In conclusion, external validation studies can readily be amended to incorporate model uncertainty, and we recommend that such studies report total variance as well as the relative share of variance components for all performance metrics.
Related
- Development and external evaluation of an interpretable machine-learning model for early prediction of organ failure in higher-risk acute pancreatitis patients: A multicentre cohort study
- CASCADE Conformal Prediction: Uncertainty-Adaptive Prediction Intervals for Two-Stage Clinical Decision Support
- Uncertainty aware prediction from incomplete clinical records: a conformal prediction study of Parkinsonian classification using C-OPN data
- Beyond Point Prediction: Artificial Representative Trees with Uncertainty
- ProbabilityBench: Model Evaluation and Calibration with Exact Ground-Truth Uncertainty
BibTeX
@misc{sadatsafavi2026evaluation,
title = {{Evaluation of clinical prediction models: Decomposing performance uncertainty into model development and validation components}},
author = {Mohsen Sadatsafavi and Paul Gustafson and Richard D. Riley},
howpublished = {arXiv},
year = {2026},
month = nov,
doi = {10.48550/arxiv.2611.03337},
eprint = {2611.03337},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2611.03337},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Evaluation of clinical prediction models: Decomposing performance uncertainty into model development and validation components
AU - Mohsen Sadatsafavi
AU - Paul Gustafson
AU - Richard D. Riley
PY - 2026
DA - 2026/11/02
PB - arXiv
DO - 10.48550/arxiv.2611.03337
UR - https://arxiv.org/abs/2611.03337
AB - There has been growing emphasis on acknowledging, quantifying, and communicating the uncertainty inherent in the output of clinical prediction models. However, allowing for model uncertainty when assessing the performance of an existing model in a new sample (i.e., external validation) has not been sufficiently explored. When validating a model that generates uncertain predictions, the uncertainty of a given performance metric is affected by both model uncertainty and the finiteness of the validation sample (validation uncertainty). For a given performance metric, we show that the total uncertainty of the performance estimate can be decomposed into the uncertainty due to the model and the uncertainty due to the validation. Also, we propose M^2, the fraction of the total variance due to model uncertainty, and V^2, the fraction of the total variance due to validation uncertainty. These measures serve as complementary scalar indices that can be implemented for common performance metrics and visualizations such as calibration plots using simple Monte Carlo algorithms. Critically, our case study shows that with the same model and the same validation sample, the contribution of uncertainty sources is highly variable across performance metrics (M^2 varying from 12% for the Brier score to 78% for the calibration slope). This indicates that stakeholders who prioritize different aspects of performance (e.g., calibration versus clinical utility) might arrive at different decisions on whether future efforts should be directed towards creating a more reliable model or procuring a larger validation sample. In conclusion, external validation studies can readily be amended to incorporate model uncertainty, and we recommend that such studies report total variance as well as the relative share of variance components for all performance metrics.
ER -