Development and internal validation of triage-stage prediction models for short-term critical outcomes among adult emergency department patients entering observation or hospital-based care: a retrospective study.
- 1Department of Emergency Medicine, The Affiliated Hospital, Southwest Medical University, Luzhou, Sichuan, China.
- 2Department of Dermatology, The Affiliated Hospital, Southwest Medical University, Luzhou, Sichuan, China.
Abstract
Background
Risk stratification at emergency department triage may help identify patients who require closer assessment or monitoring. However, prediction models that rely only on information available at the triage stage remain less well evaluated. This study aimed to develop and internally validate machine learning models based on triage-stage information for predicting short-term critical outcomes in adult emergency department patients and to interpret a representative explainable model using SHAP analysis.Methods
This retrospective observational study included 1,842 consecutive adult patients who completed standardized electronic triage and subsequently entered an emergency observation or hospital-based care pathway at a tertiary general hospital between January 2022 and December 2025. The prediction time point and landmark were defined as completion of electronic emergency triage. Only patients who were free of all outcome components at the landmark were included, and candidate predictors were restricted to routinely available information recorded no later than triage completion. The primary outcome was the first critical event occurring during the subsequent 72 h: intensive care unit admission, endotracheal intubation or invasive mechanical ventilation, cardiopulmonary resuscitation, objectively documented escalation involving an implemented resuscitation-level intervention, or in-hospital death. Additional sensitivity analyses restricted the outcome to intensive care unit admission, intubation or invasive ventilation, cardiopulmonary resuscitation, or death and refitted the model without emergency triage level. The cohort was randomly divided into a development cohort and a 30% hold-out cohort. Because the hold-out cohort had already been examined during the original analysis, it was not considered an independent validation set. The primary internal-validation analysis therefore consisted of 10 repetitions of nested cross-validation within the development cohort, using five stratified outer folds for performance estimation and four stratified inner folds for preprocessing and hyperparameter tuning. Logistic regression, random forest, support vector machine, XGBoost, and LightGBM were evaluated using the same resampling framework. The previously examined exploratory hold-out cohort was retained only for secondary exploratory analysis. Model performance was assessed using discrimination, calibration, classification metrics, Brier score, decision-curve analysis, and SHAP-based interpretation. Bootstrap optimism correction was additionally used to assess overfitting.Results
All 1,842 included patients were event-free at triage completion, and 214 experienced a short-term critical outcome during the subsequent 72-hour period, corresponding to an event rate of 11.6%. The development cohort included 1,289 patients with 150 events, and the previously examined exploratory hold-out cohort included 553 patients with 64 events. In 10 repetitions of nested cross-validation, the mean AUCs were 0.821 ± 0.012 for logistic regression, 0.817 ± 0.015 for random forest, 0.752 ± 0.018 for support vector machine, 0.831 ± 0.013 for XGBoost, and 0.833 ± 0.014 for LightGBM. XGBoost was highest ranked in 42% of repeated outer-validation results, LightGBM in 48%, and logistic regression in 10%. These primary internal-validation results supported broadly comparable performance among XGBoost, LightGBM, and logistic regression rather than consistent superiority of one algorithm. In the secondary exploratory analysis, the representative XGBoost model achieved an AUC of 0.828 in the previously examined exploratory hold-out cohort. At the development-derived threshold of 0.106, its accuracy was 0.770, sensitivity 0.656, specificity 0.785, positive predictive value 0.286, negative predictive value 0.946, F1 score 0.398, and Brier score 0.080. Its paired comparisons, sensitivity analyses, calibration, decision-curve analysis, and SHAP findings in this cohort were interpreted as exploratory.Conclusions
Repeated nested cross-validation showed broadly comparable discrimination for XGBoost, LightGBM, and logistic regression in predicting post-triage short-term critical outcomes. XGBoost was retained as a representative explainable model, whereas its performance and benchmark comparisons in the previously examined exploratory hold-out cohort should be regarded as secondary exploratory analyses. The study does not establish the superiority or independent validation of XGBoost. Evaluation in a genuinely unused temporal cohort and external prospective validation remain necessary before clinical implementation.
Related
- Development and internal validation of an exploratory clinical prediction model for MRI-defined white matter injury in moderate-to-late preterm infants: a single-center retrospective study
- Development and internal validation of a baseline prediction model for ≥10-point Oswestry Disability Index improvement by episode end among adults receiving outpatient physical therapy for low back pain : A retrospective cohort study
- Development and external validation of a risk prediction model for early peripheral intravenous catheter failure in outpatients and emergency department patients receiving consecutive infusion therapy.
- Development and internal validation of a simple risk stratification score for unplanned emergency department revisits within 24 hours after discharge: A retrospective observational study
- Development and Internal Validation of a Preintubation Prediction Model for Postintubation Hypotension in Trauma Patients Undergoing Emergency Endotracheal Intubation: A Retrospective Cohort Study
BibTeX
@article{wu2026development,
title = {{Development and internal validation of triage-stage prediction models for short-term critical outcomes among adult emergency department patients entering observation or hospital-based care: a retrospective study.}},
author = {Mingwei Wu and Yao Liu and Chengli Zhou and Jinfeng Li},
journal = {Frontiers in medicine},
year = {2026},
month = oct,
doi = {10.3389/fmed.2026.1921692},
url = {https://europepmc.org/article/MED/42908943},
note = {CC BY},
}
RIS
TY - JOUR
TI - Development and internal validation of triage-stage prediction models for short-term critical outcomes among adult emergency department patients entering observation or hospital-based care: a retrospective study.
AU - Mingwei Wu
AU - Yao Liu
AU - Chengli Zhou
AU - Jinfeng Li
PY - 2026
DA - 2026/10/07
JO - Frontiers in medicine
DO - 10.3389/fmed.2026.1921692
UR - https://europepmc.org/article/MED/42908943
AB - Background Risk stratification at emergency department triage may help identify patients who require closer assessment or monitoring. However, prediction models that rely only on information available at the triage stage remain less well evaluated. This study aimed to develop and internally validate machine learning models based on triage-stage information for predicting short-term critical outcomes in adult emergency department patients and to interpret a representative explainable model using SHAP analysis.Methods This retrospective observational study included 1,842 consecutive adult patients who completed standardized electronic triage and subsequently entered an emergency observation or hospital-based care pathway at a tertiary general hospital between January 2022 and December 2025. The prediction time point and landmark were defined as completion of electronic emergency triage. Only patients who were free of all outcome components at the landmark were included, and candidate predictors were restricted to routinely available information recorded no later than triage completion. The primary outcome was the first critical event occurring during the subsequent 72 h: intensive care unit admission, endotracheal intubation or invasive mechanical ventilation, cardiopulmonary resuscitation, objectively documented escalation involving an implemented resuscitation-level intervention, or in-hospital death. Additional sensitivity analyses restricted the outcome to intensive care unit admission, intubation or invasive ventilation, cardiopulmonary resuscitation, or death and refitted the model without emergency triage level. The cohort was randomly divided into a development cohort and a 30% hold-out cohort. Because the hold-out cohort had already been examined during the original analysis, it was not considered an independent validation set. The primary internal-validation analysis therefore consisted of 10 repetitions of nested cross-validation within the development cohort, using five stratified outer folds for performance estimation and four stratified inner folds for preprocessing and hyperparameter tuning. Logistic regression, random forest, support vector machine, XGBoost, and LightGBM were evaluated using the same resampling framework. The previously examined exploratory hold-out cohort was retained only for secondary exploratory analysis. Model performance was assessed using discrimination, calibration, classification metrics, Brier score, decision-curve analysis, and SHAP-based interpretation. Bootstrap optimism correction was additionally used to assess overfitting.Results All 1,842 included patients were event-free at triage completion, and 214 experienced a short-term critical outcome during the subsequent 72-hour period, corresponding to an event rate of 11.6%. The development cohort included 1,289 patients with 150 events, and the previously examined exploratory hold-out cohort included 553 patients with 64 events. In 10 repetitions of nested cross-validation, the mean AUCs were 0.821 ± 0.012 for logistic regression, 0.817 ± 0.015 for random forest, 0.752 ± 0.018 for support vector machine, 0.831 ± 0.013 for XGBoost, and 0.833 ± 0.014 for LightGBM. XGBoost was highest ranked in 42% of repeated outer-validation results, LightGBM in 48%, and logistic regression in 10%. These primary internal-validation results supported broadly comparable performance among XGBoost, LightGBM, and logistic regression rather than consistent superiority of one algorithm. In the secondary exploratory analysis, the representative XGBoost model achieved an AUC of 0.828 in the previously examined exploratory hold-out cohort. At the development-derived threshold of 0.106, its accuracy was 0.770, sensitivity 0.656, specificity 0.785, positive predictive value 0.286, negative predictive value 0.946, F1 score 0.398, and Brier score 0.080. Its paired comparisons, sensitivity analyses, calibration, decision-curve analysis, and SHAP findings in this cohort were interpreted as exploratory.Conclusions Repeated nested cross-validation showed broadly comparable discrimination for XGBoost, LightGBM, and logistic regression in predicting post-triage short-term critical outcomes. XGBoost was retained as a representative explainable model, whereas its performance and benchmark comparisons in the previously examined exploratory hold-out cohort should be regarded as secondary exploratory analyses. The study does not establish the superiority or independent validation of XGBoost. Evaluation in a genuinely unused temporal cohort and external prospective validation remain necessary before clinical implementation.
SN - 2296-858X
ER -