Tabular foundation models are not a shortcut to calibrated clinical risk prediction: a pre-registered simulation against the true risk
Abstract
Tabular foundation models are pretrained networks that predict risk from a table without fitting a model. Whether their risks are correct (calibrated) has been judged on real data, where true risk is unknown. In a pre-registered simulation with known true risks, we compared TabPFN-3.5, TabICLv2 and Mitra-v2 with Firth logistic regression with intercept correction (FLIC) in 58 scenarios and 1,000 random designs, then on four clinical data sets. Foundation models were not a shortcut to calibration. Their calibration slope matched FLIC's within the pre-registered margin in 36 or 37 of 48 core scenarios and was better only at n = 200 with a 3% event rate. Even 3,000 patients were often not enough: with FLIC, TabPFN-3.5 or Mitra-v2, 63%-83% of samples gave risks close to the truth in average, intercept and slope at event rates of 10% and 30%, and 35%-50% at 3%. TabICLv2 did worse (8%-39% and 1%-3%). Across the random designs, foundation models had a lower true calibration error than FLIC beyond the margin in 25%-38% and a higher one in none, though small losses were predicted at n = 3000. On real data, no model resisted a change in event rate between populations; recalibration on 500 local patients largely repaired it.
Related
- Simulation-Supervised Foundation Models for Retention Time Prediction in High-Performance Liquid Chromatography beyond Experimental Data Coverage
- Do Ray-Traced Cities Make a Better Channel Prior for an Unseen Site? A Pre-Registered Study in Simulation, and a Test on Measured Channels that Reverses It
- Covariance-Calibrated Prediction-Powered Inference
- Pattern-Calibrated Multimodal Prediction under Blockwise Missingness
- Moment-Calibrated Conformal Prediction with Side Information
BibTeX
@misc{ebrahim2026tabular,
title = {{Tabular foundation models are not a shortcut to calibrated clinical risk prediction: a pre-registered simulation against the true risk}},
author = {Ebrahim Khaled Ebrahim},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.39065},
eprint = {2610.39065},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.39065},
}
RIS
TY - UNPB
TI - Tabular foundation models are not a shortcut to calibrated clinical risk prediction: a pre-registered simulation against the true risk
AU - Ebrahim Khaled Ebrahim
PY - 2026
DA - 2026/10/28
PB - arXiv
DO - 10.48550/arxiv.2610.39065
UR - https://arxiv.org/abs/2610.39065
AB - Tabular foundation models are pretrained networks that predict risk from a table without fitting a model. Whether their risks are correct (calibrated) has been judged on real data, where true risk is unknown. In a pre-registered simulation with known true risks, we compared TabPFN-3.5, TabICLv2 and Mitra-v2 with Firth logistic regression with intercept correction (FLIC) in 58 scenarios and 1,000 random designs, then on four clinical data sets. Foundation models were not a shortcut to calibration. Their calibration slope matched FLIC's within the pre-registered margin in 36 or 37 of 48 core scenarios and was better only at n = 200 with a 3% event rate. Even 3,000 patients were often not enough: with FLIC, TabPFN-3.5 or Mitra-v2, 63%-83% of samples gave risks close to the truth in average, intercept and slope at event rates of 10% and 30%, and 35%-50% at 3%. TabICLv2 did worse (8%-39% and 1%-3%). Across the random designs, foundation models had a lower true calibration error than FLIC beyond the margin in 25%-38% and a higher one in none, though small losses were predicted at n = 3000. On real data, no model resisted a change in event rate between populations; recalibration on 500 local patients largely repaired it.
ER -