Balanced Risk Integrity Index: a stratum-level metric for evaluating clinical prediction alerts under competing risks and censoring
- 1Korea University College of Medicine
Abstract
BackgroundConventional discrimination metrics, such as the area under the receiver operating characteristic curve (AUROC), do not characterize the composition of outcomes among patients flagged as high risk by clinical prediction models. This limitation is particularly relevant in critical care, where competing deaths and incomplete follow-up can complicate the interpretation of prediction alerts. We developed the Balanced Risk Integrity Index (BRII), a stratum-level metric defined as the proportion of target events among target events, competing deaths, and early-censored observations in a flagged high-risk stratum. BRII is algebraically related to positive predictive value (PPV) and is reported alongside its component decomposition to characterize the outcome composition of flagged patients.
MethodsWe evaluated BRII using seven acute kidney injury (AKI) prediction models developed in MIMIC-IV and externally validated in MIMIC-III and eICU-CRD (116,010 patients in total). Additional simulations examined the effects of competing deaths and early censoring on BRII and conventional model performance metrics.
ResultsFor 48-hour AKI prediction in the top 10% risk stratum, BRII ranged from 0.90 to 0.96 in MIMIC-III, compared with less than 0.26 across all models in eICU-CRD, where 61.5% of evaluated patients were censored before the prediction horizon. In the eICU-CRD top 10% strata for the same target, 37.0% to 50.2% of patients were censored early, 12.2% to 15.5% had AKI, and 3.4% to 8.1% died before AKI. BRII and PPV ordered at least one model pair differently in 34 of 48 cohort-outcome-top-k groups. Under simulated random censoring for 48-hour AKI prediction, BRII declined by up to 0.80 while binary-label AUROC changed by less than 0.02.
ConclusionsBRII provided complementary information on high-risk stratum composition, yielding different model rankings from PPV and revealing substantial differences across external validation cohorts. Interpreted alongside PPV and its component decomposition, BRII offers a standardized approach to evaluating clinical prediction alerts across models and observation environments.
Related
- Competing Risk Prediction of COVID-19 Hospital Mortality with Model Diagnostics and Exploratory Evaluation in Mexico
- Evaluating Large Language Models as Tools to Navigate Researchers in Rapidly Evolving Research Landscapes: A Case Study in Cancer Drug Response Prediction
- Evaluating the impact of social determinants of health on model performance and fairness in artificial intelligence for glaucoma progression prediction
- From prediction to functional delivery: evaluating artificial intelligence in lipid nanoparticle design
- Evaluating Clinical Foundation Models for Early Alzheimer's Disease and Related Dementia Prediction from Longitudinal EHRs
BibTeX
@misc{jeong2026balanced,
title = {{Balanced Risk Integrity Index: a stratum-level metric for evaluating clinical prediction alerts under competing risks and censoring}},
author = {I. Jeong and T. Lee and J.-H. Park and Y. Kim and H. Lee},
howpublished = {medRxiv},
year = {2026},
month = oct,
doi = {10.64898/2026.10.14.26357330},
url = {https://www.medrxiv.org/content/10.64898/2026.10.14.26357330v1},
note = {CC BY},
}
RIS
TY - UNPB
TI - Balanced Risk Integrity Index: a stratum-level metric for evaluating clinical prediction alerts under competing risks and censoring
AU - I. Jeong
AU - T. Lee
AU - J.-H. Park
AU - Y. Kim
AU - H. Lee
PY - 2026
DA - 2026/10/14
PB - medRxiv
DO - 10.64898/2026.10.14.26357330
UR - https://www.medrxiv.org/content/10.64898/2026.10.14.26357330v1
AB - BackgroundConventional discrimination metrics, such as the area under the receiver operating characteristic curve (AUROC), do not characterize the composition of outcomes among patients flagged as high risk by clinical prediction models. This limitation is particularly relevant in critical care, where competing deaths and incomplete follow-up can complicate the interpretation of prediction alerts. We developed the Balanced Risk Integrity Index (BRII), a stratum-level metric defined as the proportion of target events among target events, competing deaths, and early-censored observations in a flagged high-risk stratum. BRII is algebraically related to positive predictive value (PPV) and is reported alongside its component decomposition to characterize the outcome composition of flagged patients. MethodsWe evaluated BRII using seven acute kidney injury (AKI) prediction models developed in MIMIC-IV and externally validated in MIMIC-III and eICU-CRD (116,010 patients in total). Additional simulations examined the effects of competing deaths and early censoring on BRII and conventional model performance metrics. ResultsFor 48-hour AKI prediction in the top 10% risk stratum, BRII ranged from 0.90 to 0.96 in MIMIC-III, compared with less than 0.26 across all models in eICU-CRD, where 61.5% of evaluated patients were censored before the prediction horizon. In the eICU-CRD top 10% strata for the same target, 37.0% to 50.2% of patients were censored early, 12.2% to 15.5% had AKI, and 3.4% to 8.1% died before AKI. BRII and PPV ordered at least one model pair differently in 34 of 48 cohort-outcome-top-k groups. Under simulated random censoring for 48-hour AKI prediction, BRII declined by up to 0.80 while binary-label AUROC changed by less than 0.02. ConclusionsBRII provided complementary information on high-risk stratum composition, yielding different model rankings from PPV and revealing substantial differences across external validation cohorts. Interpreted alongside PPV and its component decomposition, BRII offers a standardized approach to evaluating clinical prediction alerts across models and observation environments.
ER -