Reproducible benchmarking of four machine-learning classifiers on a large synthetic heart-disease competition dataset
- 1Department of General Surgery and Anaesthesia, Faculty of Medicine, Yarmouk University, Irbid, Jordan
Abstract
Background: Public synthetic datasets enable transparent model comparison but cannot establish performance in clinical populations. We asked how four fixed classifiers compare under one leakage-controlled validation framework on a large synthetic heart-disease competition dataset. Methods: The official training set contained 630,000 observations (44.83% outcome prevalence) and 13 predictors; the 270,000-row test set had no outcome labels. L2-regularized logistic regression, HistGradientBoosting, CatBoost, and LightGBM were evaluated using identical stratified five-fold outer splits. Logistic-regression preprocessing was fitted within outer-training data, and early stopping used outer-training data only. The primary measure was pooled out-of-fold (OOF) AUROC; secondary measures were average precision, Brier score, calibration intercept and slope, fold variability, and train–validation optimism. Duplicate/overlap, identifier-only, and permuted-outcome controls assessed leakage. Results: Pooled OOF AUROC was 0.955443 for CatBoost, 0.955324 for LightGBM, 0.955066 for HistGradientBoosting, and 0.952874 for logistic regression. Corresponding average precision ranged from 0.945912 to 0.948828 and Brier scores from 0.081116 to 0.083518. Calibration intercepts ranged from −0.001299 to 0.002153 and slopes from 0.998896 to 1.004388. Mean optimism was 0.000017–0.002686. No duplicated identifiers, duplicated feature profiles, or train–test feature-profile overlap were found; identifier-only AUROC was 0.500121 and mean permuted-outcome AUROC was 0.500615. Conclusions: All four models discriminated strongly and showed close internal calibration, while gradient-boosting models had only marginally higher AUROC than logistic regression. The principal contribution is a fully rerunnable, controlled benchmark rather than a novel algorithm or a clinically validated prediction model. External performance in real clinical cohorts remains unknown.
Related
- A mechanism-guided simulation framework for synthetic tax-risk data: Constraint validation and reproducible benchmarking
- Benchmarking Goodness-of-Fit and Calibration Algorithms for Logistic Regression Classifiers: A Large-Scale Simulation Study under Sparse Data
- Seq2Synth: Benchmarking Temporal Fidelity in Synthetic Sequential Tabular Data
- A probability calibrator for machine-learning classifiers
- A Reproducible Statistical-Learning Framework for Sparse Time-Indexed Transaction Data
BibTeX
@article{ennab2026reproducible,
title = {{Reproducible benchmarking of four machine-learning classifiers on a large synthetic heart-disease competition dataset}},
author = {Raed M. Ennab},
journal = {PLOS One},
year = {2026},
month = sep,
doi = {10.1371/journal.pone.0354304},
url = {https://doi.org/10.1371/journal.pone.0354304},
note = {CC BY 4.0},
}
RIS
TY - JOUR
TI - Reproducible benchmarking of four machine-learning classifiers on a large synthetic heart-disease competition dataset
AU - Raed M. Ennab
PY - 2026
DA - 2026/09/21
JO - PLOS One
DO - 10.1371/journal.pone.0354304
UR - https://doi.org/10.1371/journal.pone.0354304
AB - Background: Public synthetic datasets enable transparent model comparison but cannot establish performance in clinical populations. We asked how four fixed classifiers compare under one leakage-controlled validation framework on a large synthetic heart-disease competition dataset. Methods: The official training set contained 630,000 observations (44.83% outcome prevalence) and 13 predictors; the 270,000-row test set had no outcome labels. L2-regularized logistic regression, HistGradientBoosting, CatBoost, and LightGBM were evaluated using identical stratified five-fold outer splits. Logistic-regression preprocessing was fitted within outer-training data, and early stopping used outer-training data only. The primary measure was pooled out-of-fold (OOF) AUROC; secondary measures were average precision, Brier score, calibration intercept and slope, fold variability, and train–validation optimism. Duplicate/overlap, identifier-only, and permuted-outcome controls assessed leakage. Results: Pooled OOF AUROC was 0.955443 for CatBoost, 0.955324 for LightGBM, 0.955066 for HistGradientBoosting, and 0.952874 for logistic regression. Corresponding average precision ranged from 0.945912 to 0.948828 and Brier scores from 0.081116 to 0.083518. Calibration intercepts ranged from −0.001299 to 0.002153 and slopes from 0.998896 to 1.004388. Mean optimism was 0.000017–0.002686. No duplicated identifiers, duplicated feature profiles, or train–test feature-profile overlap were found; identifier-only AUROC was 0.500121 and mean permuted-outcome AUROC was 0.500615. Conclusions: All four models discriminated strongly and showed close internal calibration, while gradient-boosting models had only marginally higher AUROC than logistic regression. The principal contribution is a fully rerunnable, controlled benchmark rather than a novel algorithm or a clinically validated prediction model. External performance in real clinical cohorts remains unknown.
SN - 1932-6203
ER -