Estimating Stability of Individual-Specific Predictions from AI-based prediction models using Bootstrap and Cross-validation Resampling
Abstract
Clinical prediction models are typically developed in a single finite dataset, which is one of many possible datasets that could have been collected. Applying the same model development procedure in another sample will provide different predictions for an individual. The degree to which an individual's prediction varies across development sets is referred to as prediction instability and is not captured by average performance measures. Resampling of the training data offers a practical way to assess instability, but its ability to do so accurately and reliably for different model choices is unclear. In a simulation study, we considered non-parametric, parametric, and out-of-bag bootstrap methods and repeated 10-, 5-, and 2-fold cross-validation (CV) to estimate metrics of prediction instability for Elastic Net logistic regression and Random Forest. The same metrics estimated in independent development samples provided the reference. We assessed individual-level prediction variance, empirical 95% range width, and mean absolute differences from the original prediction and true outcome probability. Instability increased in smaller development samples, was greater for Random Forest, and varied across the probability range. Repeated 5- and 10-fold CV underestimated prediction instability across scenarios. Default non-parametric bootstrap more closely approximated independent sampling for Elastic Net but often overestimated instability for Random Forest, as did parametric bootstrap. Out-of-bag bootstrap and repeated 2-fold CV generally agreed most closely with independent sampling, although no procedure was uniformly best. Resampling procedures should not be used interchangeably, nor should less variable predictions be taken as evidence of greater stability. To estimate prediction instability, prefer procedures that predict on held-out data and limit overlap between training resamples.
Related
- Beyond Point Predictions: Distribution-Informed Prediction-Powered Inference
- Reconstructing the Human Development Index at the Individual Level from Survey Microdata: Development and Validation of a Methodology
- Local conformal prediction for individual causal effects
- Translation of Black-Box Clinical Prediction Models into Standalone Transparent Nomograms: Temporal External Validation in Heart Transplantation
- Prediction-Powered Smoothing and Validation for Disaggregated AI Evaluation
BibTeX
@misc{carriero2026estimating,
title = {{Estimating Stability of Individual-Specific Predictions from AI-based prediction models using Bootstrap and Cross-validation Resampling}},
author = {Alex Carriero and Martijn L. G. van Dam and Anne A. H. de Hond and Richard D. Riley and Gary S. Collins and Karel G. M. Moons and Maarten van Smeden and Nicole S. Erler},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.33744},
eprint = {2610.33744},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.33744},
note = {CC BY-NC-ND 4.0},
}
RIS
TY - UNPB
TI - Estimating Stability of Individual-Specific Predictions from AI-based prediction models using Bootstrap and Cross-validation Resampling
AU - Alex Carriero
AU - Martijn L. G. van Dam
AU - Anne A. H. de Hond
AU - Richard D. Riley
AU - Gary S. Collins
AU - Karel G. M. Moons
AU - Maarten van Smeden
AU - Nicole S. Erler
PY - 2026
DA - 2026/10/24
PB - arXiv
DO - 10.48550/arxiv.2610.33744
UR - https://arxiv.org/abs/2610.33744
AB - Clinical prediction models are typically developed in a single finite dataset, which is one of many possible datasets that could have been collected. Applying the same model development procedure in another sample will provide different predictions for an individual. The degree to which an individual's prediction varies across development sets is referred to as prediction instability and is not captured by average performance measures. Resampling of the training data offers a practical way to assess instability, but its ability to do so accurately and reliably for different model choices is unclear. In a simulation study, we considered non-parametric, parametric, and out-of-bag bootstrap methods and repeated 10-, 5-, and 2-fold cross-validation (CV) to estimate metrics of prediction instability for Elastic Net logistic regression and Random Forest. The same metrics estimated in independent development samples provided the reference. We assessed individual-level prediction variance, empirical 95% range width, and mean absolute differences from the original prediction and true outcome probability. Instability increased in smaller development samples, was greater for Random Forest, and varied across the probability range. Repeated 5- and 10-fold CV underestimated prediction instability across scenarios. Default non-parametric bootstrap more closely approximated independent sampling for Elastic Net but often overestimated instability for Random Forest, as did parametric bootstrap. Out-of-bag bootstrap and repeated 2-fold CV generally agreed most closely with independent sampling, although no procedure was uniformly best. Resampling procedures should not be used interchangeably, nor should less variable predictions be taken as evidence of greater stability. To estimate prediction instability, prefer procedures that predict on held-out data and limit overlap between training resamples.
ER -