Evaluating uncertainty for detecting performance loss and guiding deferral after model transfer between hospital systems
- 1Charité – Universitätsmedizin Berlin
Abstract
Clinical prediction models often perform worse after transfer between hospital systems, but deterioration becomes apparent only once outcomes are known. We investigated whether uncertainty measures available at prediction time warn of performance loss and can help identify individual predictions requiring review. Three neural network architectures were trained on MIMIC-IV for in-hospital mortality prediction and evaluated on eICU-CRD. Measures included predictive entropy, mutual information from Monte Carlo dropout and deep ensembles, and representation density. After transfer, AUROC fell from 0.86-0.87 to 0.76-0.77 and calibration worsened, while entropy and mutual information were lower. None of the measures reliably indicated the reduced performance. Deferring the 20% with the highest entropy reduced the remaining error rate from 17.5-20.1% to 6.3-8.0%; the deferred patients included 91-99% of false alarms but only 2-7% of missed deaths, which carried low uncertainty. Uncertainty therefore supported selective review but failed to reveal performance loss.
Related
- Diagnostic Performance of Clinical Examination for Detecting Knee Effusion: Impact of Obesity and Periarticular Adiposity
- Diagnostic Performance of Color Doppler Ultrasound for Detecting Nuchal Cord and Associated Immediate Neonatal Outcomes at Term in Uganda: A Prospective Diagnostic‐Accuracy Cohort Study
- Diagnostic Performance of a Deep Learning Model Using Phonocardiogram and Single‐Lead ECG Signals for Detecting Moderate‐or‐Greater Aortic Stenosis: Multicenter External Validation
- A picture fuzzy SWARA–VIKOR framework for evaluating drone-swarm navigation and mission-coordination strategies under uncertainty
- Comprehensive Performance Verification of the Mindray MN2880 System in Detecting Respiratory Virus Nucleic Acids
BibTeX
@misc{behling2026evaluating,
title = {{Evaluating uncertainty for detecting performance loss and guiding deferral after model transfer between hospital systems}},
author = {S. Behling and M. Schulte-Althoff and S. Haufe and E. Grünewald and F. Balzer and N. Frey and S. D. Boie},
howpublished = {medRxiv},
year = {2026},
month = nov,
doi = {10.64898/2026.11.03.26367064},
url = {https://www.medrxiv.org/content/10.64898/2026.11.03.26367064v1},
note = {CC BY},
}
RIS
TY - UNPB
TI - Evaluating uncertainty for detecting performance loss and guiding deferral after model transfer between hospital systems
AU - S. Behling
AU - M. Schulte-Althoff
AU - S. Haufe
AU - E. Grünewald
AU - F. Balzer
AU - N. Frey
AU - S. D. Boie
PY - 2026
DA - 2026/11/03
PB - medRxiv
DO - 10.64898/2026.11.03.26367064
UR - https://www.medrxiv.org/content/10.64898/2026.11.03.26367064v1
AB - Clinical prediction models often perform worse after transfer between hospital systems, but deterioration becomes apparent only once outcomes are known. We investigated whether uncertainty measures available at prediction time warn of performance loss and can help identify individual predictions requiring review. Three neural network architectures were trained on MIMIC-IV for in-hospital mortality prediction and evaluated on eICU-CRD. Measures included predictive entropy, mutual information from Monte Carlo dropout and deep ensembles, and representation density. After transfer, AUROC fell from 0.86-0.87 to 0.76-0.77 and calibration worsened, while entropy and mutual information were lower. None of the measures reliably indicated the reduced performance. Deferring the 20% with the highest entropy reduced the remaining error rate from 17.5-20.1% to 6.3-8.0%; the deferred patients included 91-99% of false alarms but only 2-7% of missed deaths, which carried low uncertainty. Uncertainty therefore supported selective review but failed to reveal performance loss.
ER -