Calibration, Uncertainty, and Data Efficiency of Deep Learning Chest Radiograph Classifiers Under Cross-Dataset Distribution Shift: A Multi-Dataset Benchmark
- 1KIST Medical College and Teaching Hospital
Abstract
PurposeTo evaluate the external transportability of source-derived probability calibration for chest radiograph deep learning classifiers under distribution shift and determine its dependence on sample size and disease prevalence.
Materials and MethodsThis retrospective secondary computational analysis of public, de-identified radiographic datasets involved no participant contact or new data collection. Cohort sizes were prespecified by availability without prospective power calculation: 106,686 NIH ChestX-ray14 images (29,323 patients; 1992-2015) for development and internal testing, 20,000 CheXpert images (20,000 patients; October 2002-July 2017), and 14,998 VinBigData images (2018-2020) for external testing. ResNet-18, DenseNet-121, and DeiT-Tiny classifiers for cardiomegaly and pleural effusion were trained across four patient fractions and three seeds (36 fits). Calibrators were fitted exclusively on NIH data. Twelve primary external Brier-score contrasts (temperature-scaled versus raw probabilities) were evaluated using paired cluster-bootstrapping with Holm adjustment.
ResultsTemperature scaling reduced Brier scores in nine of 12 primary contrasts but worsened pleural effusion calibration on VinBigData across all architectures (Brier increases: ResNet-18, 0.001460 [95% CI: 0.001347, 0.001568]; DenseNet-121, 0.001193 [95% CI: 0.001109, 0.001274]; DeiT-Tiny, 0.000498 [95% CI: 0.000417, 0.000577]; all adjusted P=.006). Standardizing class weights to NIH prevalence reversed all CheXpert improvements. Although full versus 10% training improved external AUROC across all comparisons, DeiT-Tiny CheXpert pleural effusion Brier score worsened by 0.007039 despite a 0.032466 AUROC gain. At 80% selective image retention, calibrated ensembles retained only 12.3%-17.2% of VinBigData positive pleural effusion examinations.
ConclusionSource-fitted calibration is not uniformly transportable across health systems, and discrimination gains do not guarantee calibrated probabilities. External evaluation must simultaneously assess probability reliability, prevalence shifts, and retained-case composition.
Related
- Phantom Fairness: A Reproducible Audit of How Automatically Extracted Labels Can Conceal Demographic Disparities in Chest Radiograph Classifiers
- Calibrated per-pixel uncertainty for low-dose paediatric chest-radiograph denoising at no fidelity cost
- Non-generative readout with few-label calibration versus LoRA fine-tuning of a large language model for ASA physical status classification: accuracy, speed, cost and label efficiency in three data sources
- A hybrid predictive and uncertainty-aware framework for renewable hydrogen system efficiency assessment and sustainable energy planning
- Uncertainty-aware bilevel energy storage planning for active distribution networks: A fixed-topology DistFlow-based MISOCP framework with voltage-security assessment
BibTeX
@misc{shamim2026calibration,
title = {{Calibration, Uncertainty, and Data Efficiency of Deep Learning Chest Radiograph Classifiers Under Cross-Dataset Distribution Shift: A Multi-Dataset Benchmark}},
author = {F. Shamim and R. A. Hussain},
howpublished = {medRxiv},
year = {2026},
month = oct,
doi = {10.64898/2026.10.14.26365516},
url = {https://www.medrxiv.org/content/10.64898/2026.10.14.26365516v1},
note = {CC BY-NC},
}
RIS
TY - UNPB
TI - Calibration, Uncertainty, and Data Efficiency of Deep Learning Chest Radiograph Classifiers Under Cross-Dataset Distribution Shift: A Multi-Dataset Benchmark
AU - F. Shamim
AU - R. A. Hussain
PY - 2026
DA - 2026/10/14
PB - medRxiv
DO - 10.64898/2026.10.14.26365516
UR - https://www.medrxiv.org/content/10.64898/2026.10.14.26365516v1
AB - PurposeTo evaluate the external transportability of source-derived probability calibration for chest radiograph deep learning classifiers under distribution shift and determine its dependence on sample size and disease prevalence. Materials and MethodsThis retrospective secondary computational analysis of public, de-identified radiographic datasets involved no participant contact or new data collection. Cohort sizes were prespecified by availability without prospective power calculation: 106,686 NIH ChestX-ray14 images (29,323 patients; 1992-2015) for development and internal testing, 20,000 CheXpert images (20,000 patients; October 2002-July 2017), and 14,998 VinBigData images (2018-2020) for external testing. ResNet-18, DenseNet-121, and DeiT-Tiny classifiers for cardiomegaly and pleural effusion were trained across four patient fractions and three seeds (36 fits). Calibrators were fitted exclusively on NIH data. Twelve primary external Brier-score contrasts (temperature-scaled versus raw probabilities) were evaluated using paired cluster-bootstrapping with Holm adjustment. ResultsTemperature scaling reduced Brier scores in nine of 12 primary contrasts but worsened pleural effusion calibration on VinBigData across all architectures (Brier increases: ResNet-18, 0.001460 [95% CI: 0.001347, 0.001568]; DenseNet-121, 0.001193 [95% CI: 0.001109, 0.001274]; DeiT-Tiny, 0.000498 [95% CI: 0.000417, 0.000577]; all adjusted P=.006). Standardizing class weights to NIH prevalence reversed all CheXpert improvements. Although full versus 10% training improved external AUROC across all comparisons, DeiT-Tiny CheXpert pleural effusion Brier score worsened by 0.007039 despite a 0.032466 AUROC gain. At 80% selective image retention, calibrated ensembles retained only 12.3%-17.2% of VinBigData positive pleural effusion examinations. ConclusionSource-fitted calibration is not uniformly transportable across health systems, and discrimination gains do not guarantee calibrated probabilities. External evaluation must simultaneously assess probability reliability, prevalence shifts, and retained-case composition.
ER -