External Evaluation and Calibration Drift of Explainable Machine Learning Models for Acute and Chronic GVHD Prediction After Allogeneic HSCT
- 1Cleveland Clinic Abu Dhabi
Abstract
ObjectivesMachine learning models for graft-versus-host disease (GVHD) after allogeneic hematopoietic stem cell transplantation (allo-HSCT) are rarely evaluated externally or for calibration. We developed explainable models for grade II-IV acute and moderate-to-severe chronic GVHD and evaluated their transportability across multiple cohorts.
Materials and methodsGradient boosting models were trained on three CIBMTR registry-derived datasets (N = 2,509) and externally evaluated in six cohorts (N = 14,788), including a Middle Eastern cohort. We assessed discrimination (AUROC), calibration (logistic recalibration intercept and slope; Brier score), and SHAP feature explanations.
ResultsInternal discrimination was higher for chronic (AUROC 0.736) than acute GVHD (0.624); chronic calibration was good (intercept +0.258, slope 1.073), whereas the acute model was underdispersed (slope 1.802). Externally, the chronic model retained modest, above-chance discrimination (AUROC 0.54-0.64), whereas acute performance fell toward chance (0.52-0.57). Calibration drifted unpredictably, overestimating risk in 9 of 10 evaluable cohort-outcome combinations (intercepts -0.78 to +0.15; slopes 0.12 to 1.98; Brier 0.23-0.27).
DiscussionRoutine registry variables carry a transportable signal for chronic GVHD, but acute GVHD likely requires additional biological or clinical predictors. Neither the direction nor the magnitude of calibration drift could be anticipated from training performance.
ConclusionInternal performance does not guarantee external utility. Cohort-specific recalibration and routine calibration reporting are required before any clinical use; this study offers a reusable template for multi-cohort external evaluation of clinical prediction models.
Lay SummaryAfter a stem cell transplant from a donor, the donors immune cells can attack the patients body, a complication called graft-versus-host disease (GVHD). It can appear early (acute) or months later (chronic), and knowing who is at highest risk before transplant could help doctors plan prevention and follow-up. We built computer models that estimate GVHD risk using information routinely recorded at transplant, trained them on 2,509 patients from a large US transplant registry, and then tested them on six other groups totalling almost 15,000 patients, including patients from the United Arab Emirates and Jordan. The model for chronic GVHD kept a modest ability to separate higher-from lower-risk patients in every group, but the model for acute GVHD performed little better than chance outside the data it was built on. In almost every new group, both models overestimated how many patients would develop GVHD, by amounts that could not be predicted in advance. This shows that a model that works well where it was developed may not work elsewhere, and that its risk estimates must be checked and adjusted locally before it is ever used to guide care.
Related
- Perioperative Intravenous Antibiotic Regimens and Surgical Site Infection After Colorectal Surgery: Clinical Associations and External Evaluation of Risk Prediction Models
- Integrated transcriptomic and clinical prediction model for malignant pulmonary nodules : Model development, external dataset evaluation, and single-center expression validation
- Development and external evaluation of an interpretable machine-learning model for early prediction of organ failure in higher-risk acute pancreatitis patients: A multicentre cohort study
- Single-Center External Validation and Calibration of the ACS NSQIP Surgical Risk Calculator in Emergency Colorectal Cancer Surgery
- Explainable machine learning and multivariable logistic regression for identifying prevalent self-reported physician-diagnosed chronic lung disease in postmenopausal women: development and multi-cohort external validation
BibTeX
@misc{syed2026external,
title = {{External Evaluation and Calibration Drift of Explainable Machine Learning Models for Acute and Chronic GVHD Prediction After Allogeneic HSCT}},
author = {N. Syed and N. Ahmed and M. Abuhaleeqa and F. M. Al Kaabi and A. Raza and A. Al Zaki and F. Sammour and Y. Alkhatib and D. Gopalakrishnan and I. Afrooz and M. Damlaj and H. Abu Jazar and H. Abdel-Razeq and K. Halahleh and M. Yaqub and S. Hashmi},
howpublished = {medRxiv},
year = {2026},
month = jun,
doi = {10.64898/2026.06.14.26355639},
url = {https://www.medrxiv.org/content/10.64898/2026.06.14.26355639v2},
note = {CC BY-NC-ND},
}
RIS
TY - UNPB
TI - External Evaluation and Calibration Drift of Explainable Machine Learning Models for Acute and Chronic GVHD Prediction After Allogeneic HSCT
AU - N. Syed
AU - N. Ahmed
AU - M. Abuhaleeqa
AU - F. M. Al Kaabi
AU - A. Raza
AU - A. Al Zaki
AU - F. Sammour
AU - Y. Alkhatib
AU - D. Gopalakrishnan
AU - I. Afrooz
AU - M. Damlaj
AU - H. Abu Jazar
AU - H. Abdel-Razeq
AU - K. Halahleh
AU - M. Yaqub
AU - S. Hashmi
PY - 2026
DA - 2026/06/14
PB - medRxiv
DO - 10.64898/2026.06.14.26355639
UR - https://www.medrxiv.org/content/10.64898/2026.06.14.26355639v2
AB - ObjectivesMachine learning models for graft-versus-host disease (GVHD) after allogeneic hematopoietic stem cell transplantation (allo-HSCT) are rarely evaluated externally or for calibration. We developed explainable models for grade II-IV acute and moderate-to-severe chronic GVHD and evaluated their transportability across multiple cohorts. Materials and methodsGradient boosting models were trained on three CIBMTR registry-derived datasets (N = 2,509) and externally evaluated in six cohorts (N = 14,788), including a Middle Eastern cohort. We assessed discrimination (AUROC), calibration (logistic recalibration intercept and slope; Brier score), and SHAP feature explanations. ResultsInternal discrimination was higher for chronic (AUROC 0.736) than acute GVHD (0.624); chronic calibration was good (intercept +0.258, slope 1.073), whereas the acute model was underdispersed (slope 1.802). Externally, the chronic model retained modest, above-chance discrimination (AUROC 0.54-0.64), whereas acute performance fell toward chance (0.52-0.57). Calibration drifted unpredictably, overestimating risk in 9 of 10 evaluable cohort-outcome combinations (intercepts -0.78 to +0.15; slopes 0.12 to 1.98; Brier 0.23-0.27). DiscussionRoutine registry variables carry a transportable signal for chronic GVHD, but acute GVHD likely requires additional biological or clinical predictors. Neither the direction nor the magnitude of calibration drift could be anticipated from training performance. ConclusionInternal performance does not guarantee external utility. Cohort-specific recalibration and routine calibration reporting are required before any clinical use; this study offers a reusable template for multi-cohort external evaluation of clinical prediction models. Lay SummaryAfter a stem cell transplant from a donor, the donors immune cells can attack the patients body, a complication called graft-versus-host disease (GVHD). It can appear early (acute) or months later (chronic), and knowing who is at highest risk before transplant could help doctors plan prevention and follow-up. We built computer models that estimate GVHD risk using information routinely recorded at transplant, trained them on 2,509 patients from a large US transplant registry, and then tested them on six other groups totalling almost 15,000 patients, including patients from the United Arab Emirates and Jordan. The model for chronic GVHD kept a modest ability to separate higher-from lower-risk patients in every group, but the model for acute GVHD performed little better than chance outside the data it was built on. In almost every new group, both models overestimated how many patients would develop GVHD, by amounts that could not be predicted in advance. This shows that a model that works well where it was developed may not work elsewhere, and that its risk estimates must be checked and adjusted locally before it is ever used to guide care.
ER -