Temporal leakage and evaluation-boundary effects in heart failure mortality prediction: a unified reanalysis of the 299-patient heart failure clinical records dataset
- 1Graduate School of Heilongjiang University of Traditional Chinese Medicine
- 2Department of Cardiology III, First Affiliated Hospital of Heilongjiang University of Traditional Chinese Medicine
Abstract
Background: The 299-patient Heart Failure Clinical Records dataset is widely used for mortality prediction, yet reported performance varies substantially. Differences in prediction time, follow-up handling, resampling, and data splitting may materially affect apparent model performance.
Objective: To examine how prediction-time alignment, modelling procedures, and evaluation boundaries influence performance estimates on this shared benchmark, using a targeted methodological audit and harmonized reanalysis of representative modelling configurations.
Methods: Four representative modelling routes were evaluated using the same dataset. Admission-time classification used 11 baseline variables and stratified 10-fold cross-validation, with applicable resampling and feature-construction procedures restricted to the training data. A parsimonious random forest using ejection fraction and serum creatinine was also assessed. Follow-up duration was excluded from baseline classification and retained for time-to-event analyses. Sensitivity analyses examined follow-up-duration inclusion and changes in preprocessing boundaries. Patient-level bootstrap confidence intervals, repeated-seed analyses, calibration, and exploratory decision-curve analysis were additionally performed.
Results: Fold-mean AUROCs across the four 11-variable classification configurations ranged from 0.724 to 0.784. The two-feature random forest achieved a pooled out-of-fold AUROC of 0.779 (95% CI, 0.721–0.832), with a paired difference of 0.007 (95% CI, −0.047 to 0.062) from the 11-variable random-forest comparator. Adding follow-up duration increased fold-mean AUROC by 0.121–0.135 within matched predictor sets. The two-feature model retained substantial discrimination but showed limited calibration, with a calibration slope of 0.234 (95% CI, 0.154–0.404). Preprocessing sensitivity analyses also produced substantial variation in apparent performance.
Conclusions: Prediction-time alignment and evaluation procedures materially influenced performance estimates on this benchmark. Similar discrimination did not imply comparable probability calibration or equivalent patient-level risk estimates. Clinical interpretation therefore requires explicit definition of prediction time, outcome horizon, model implementation, and evaluation procedures, followed by validation in independent contemporary cohorts.
Related
- On-Device ECG and Vital-Sign Prediction of 24-Hour Mortality and Unplanned ICU Admission in Emergency Department Patients: A Retrospective Evaluation of MEDI-AI
- Competing Risk Prediction of COVID-19 Hospital Mortality with Model Diagnostics and Exploratory Evaluation in Mexico
- Temporal interpretation of physiologic group contributions for ICU mortality prediction
- Temporal Validation and Simplification of Mortality Prediction for Perioperative Cardiopulmonary Resuscitation
- Incremental value of bilateral ankle-brachial index for ten-year heart and cerebrovascular mortality prediction: an exploratory temporal benchmark in NHANES
BibTeX
@article{diao2026temporal,
title = {{Temporal leakage and evaluation-boundary effects in heart failure mortality prediction: a unified reanalysis of the 299-patient heart failure clinical records dataset}},
author = {Yinuo Diao and Kuikui Zhang and Wenzeng Wang},
journal = {Frontiers in Cardiovascular Medicine},
year = {2026},
month = nov,
doi = {10.3389/fcvm.2026.1980349},
url = {https://www.frontiersin.org/articles/10.3389/fcvm.2026.1980349/full},
note = {CC BY 4.0},
}
RIS
TY - JOUR
TI - Temporal leakage and evaluation-boundary effects in heart failure mortality prediction: a unified reanalysis of the 299-patient heart failure clinical records dataset
AU - Yinuo Diao
AU - Kuikui Zhang
AU - Wenzeng Wang
PY - 2026
DA - 2026/11/16
JO - Frontiers in Cardiovascular Medicine
DO - 10.3389/fcvm.2026.1980349
UR - https://www.frontiersin.org/articles/10.3389/fcvm.2026.1980349/full
AB - Background: The 299-patient Heart Failure Clinical Records dataset is widely used for mortality prediction, yet reported performance varies substantially. Differences in prediction time, follow-up handling, resampling, and data splitting may materially affect apparent model performance. Objective: To examine how prediction-time alignment, modelling procedures, and evaluation boundaries influence performance estimates on this shared benchmark, using a targeted methodological audit and harmonized reanalysis of representative modelling configurations. Methods: Four representative modelling routes were evaluated using the same dataset. Admission-time classification used 11 baseline variables and stratified 10-fold cross-validation, with applicable resampling and feature-construction procedures restricted to the training data. A parsimonious random forest using ejection fraction and serum creatinine was also assessed. Follow-up duration was excluded from baseline classification and retained for time-to-event analyses. Sensitivity analyses examined follow-up-duration inclusion and changes in preprocessing boundaries. Patient-level bootstrap confidence intervals, repeated-seed analyses, calibration, and exploratory decision-curve analysis were additionally performed. Results: Fold-mean AUROCs across the four 11-variable classification configurations ranged from 0.724 to 0.784. The two-feature random forest achieved a pooled out-of-fold AUROC of 0.779 (95% CI, 0.721–0.832), with a paired difference of 0.007 (95% CI, −0.047 to 0.062) from the 11-variable random-forest comparator. Adding follow-up duration increased fold-mean AUROC by 0.121–0.135 within matched predictor sets. The two-feature model retained substantial discrimination but showed limited calibration, with a calibration slope of 0.234 (95% CI, 0.154–0.404). Preprocessing sensitivity analyses also produced substantial variation in apparent performance. Conclusions: Prediction-time alignment and evaluation procedures materially influenced performance estimates on this benchmark. Similar discrimination did not imply comparable probability calibration or equivalent patient-level risk estimates. Clinical interpretation therefore requires explicit definition of prediction time, outcome horizon, model implementation, and evaluation procedures, followed by validation in independent contemporary cohorts.
SN - 2297-055X
ER -