Do stable performance metrics guarantee stable model predictions?: An empirical investigation using the GUSTO-1 dataset
Abstract
Model validation is routinely used to quantify and correct for overfitting in performance estimates, but it does not fully address the stability of models developed from slightly different training samples. We investigated whether stable discrimination implies stable individual predictions and compared logistic regression (LR) with artificial neural networks (ANNs). Using data from 40,830 participants in the GUSTO-I trial, with 30-day mortality as the outcome, we created six sample-size scenarios with varying events per variable (EPV). In each scenario, LR and ANN models were developed using eight predictors, and 200 bootstrap resamples were used to assess stability in discrimination, mean absolute prediction error (MAPE), and the classification instability index (CII). For LR, discrimination stabilized at an EPV of 44.50, whereas MAPE and CII did not stabilize until an EPV of 89.12. ANN discrimination was more volatile, with optimism stabilizing only at an EPV of 178.25, and ANN predictions were severely miscalibrated at an EPV of 4.50. Overall, discrimination stabilized earlier than individual predictions for both approaches, with greater instability and larger sample-size requirements for ANN. Model developers should therefore assess and report prediction-level stability, because discrimination metrics alone may not reflect the reliability of individual predictions.
Related
- Comparative Performance Analysis of Optimization Solvers for Linear Support Vector Machines: Impact of Preconditioning and Dimensionality An Empirical Study Using the Benchopt Framework
- Comment on Vogels et al.'s (2024) "Bayesian Structure Learning in Undirected Gaussian Graphical Models: Literature Review with Empirical Comparison": An updated performance check using BGGM
- Hyper-Heuristics and Scheduling Problems: Strategies, Application Areas, and Performance Metrics
- Bayes Estimators with Performance Comparable to Empirical Bayes Estimators and Improved Local Robustness
- Hit Selection Using SSMD-Based Machine Learning Performance Metrics in High-Throughput Screening Assays
BibTeX
@misc{isaradech2026stable,
title = {{Do stable performance metrics guarantee stable model predictions?: An empirical investigation using the GUSTO-1 dataset}},
author = {Natthanaphop Isaradech and Wachiranun Sirikul and Wuttipat Kiratipaisarl and Pakpoom Wongyikul and Noraworn Jirattikanwong and Phichayut Phinyo},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2609.19019},
eprint = {2609.19019},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2609.19019},
note = {CC BY-NC-ND 4.0},
}
RIS
TY - UNPB
TI - Do stable performance metrics guarantee stable model predictions?: An empirical investigation using the GUSTO-1 dataset
AU - Natthanaphop Isaradech
AU - Wachiranun Sirikul
AU - Wuttipat Kiratipaisarl
AU - Pakpoom Wongyikul
AU - Noraworn Jirattikanwong
AU - Phichayut Phinyo
PY - 2026
DA - 2026/09/01
PB - arXiv
DO - 10.48550/arxiv.2609.19019
UR - https://arxiv.org/abs/2609.19019
AB - Model validation is routinely used to quantify and correct for overfitting in performance estimates, but it does not fully address the stability of models developed from slightly different training samples. We investigated whether stable discrimination implies stable individual predictions and compared logistic regression (LR) with artificial neural networks (ANNs). Using data from 40,830 participants in the GUSTO-I trial, with 30-day mortality as the outcome, we created six sample-size scenarios with varying events per variable (EPV). In each scenario, LR and ANN models were developed using eight predictors, and 200 bootstrap resamples were used to assess stability in discrimination, mean absolute prediction error (MAPE), and the classification instability index (CII). For LR, discrimination stabilized at an EPV of 44.50, whereas MAPE and CII did not stabilize until an EPV of 89.12. ANN discrimination was more volatile, with optimism stabilizing only at an EPV of 178.25, and ANN predictions were severely miscalibrated at an EPV of 4.50. Overall, discrimination stabilized earlier than individual predictions for both approaches, with greater instability and larger sample-size requirements for ANN. Model developers should therefore assess and report prediction-level stability, because discrimination metrics alone may not reflect the reliability of individual predictions.
ER -