Calibrating Machine Learning Approaches for Probability Estimation Without Calibration Data.
7 affiliations
- 1Cardio-CARE, Medizincampus Davos, Davos, Switzerland.
- 2Department of Cardiology, University Heart and Vascular Center Hamburg, University Medical Center Hamburg-Eppendorf, Hamburg, Germany.
- 3Center for Population Health Innovation (POINT), University Heart and Vascular Center Hamburg, University Medical Center Hamburg-Eppendorf, Hamburg, Germany.
- 4Partner Site Hamburg/Kiel/Lübeck, German Center for Cardiovascular Research (DZHK), Hamburg, Germany.
- 5BDH-Klinik Elzach, Elzach, Germany.
- 6Institute for Medical Informatics, Biometry and Epidemiology, University Hospital, University Duisburg-Essen, Essen, Germany.
- 7Discipline of Statistics, School of Agriculture and Science, University of KwaZulu-Natal, Pietermaritzburg, South Africa.
Abstract
Statistical prediction models for binary outcomes are becoming increasingly popular. One significant challenge is calibrating these models to suit the characteristics of a target population that is structurally different from the original population. Calibration is especially challenging when there is no training data available from the target population. To address this problem, we propose a novel calibration method, SimCal, which uses synthetic data generated from the model development data in conjunction with marginal statistics from the calibration cohort. We show that expert judgment modeling (EJM) may be used for calibration if cross-sectional data from the target population are available comprising expert judgments about the potential outcome and the covariates. We describe three alternative calibration approaches when calibration data are lacking: similarity-binning averaging (SBA), adaptive calibration of predictions (ACP), and Elkan calibration. In a simulation study, we compare SBA, ACP, Elkan calibration, and SimCal. R code for applying these methods is provided from the re-analysis of data on coronary artery disease. We illustrate all 5 calibration approaches with a real data set for predicting functional outcome after stroke and all approaches but EJM in the re-analysis of the Cleveland Clinic data. None of the approaches performed convincingly well in all situations. SimCal performed well when model parameters were correctly specified. EJM failed on the stroke data. Further research is urgently required for calibration in the absence of calibration data.
Related
- A Hybrid Spatial Statistical Learning Framework for Individualized Probability Estimation: Application to Multisite Autism Neuroimaging
- Native Probability Outputs for Clinical Triage: An Evaluation of Calibration and Decision Accuracy
- Robust estimation in generalized linear models based on the normal quantiles of the probability integral transformation
- A Flexible Heavy-Tailed Probability Model for Skewed Positive Data: Theory, Estimation, and Empirical Assessment
- Black-Box Membership Inference via Word-Level Probability Estimation
BibTeX
@article{carluccio2026calibrating,
title = {{Calibrating Machine Learning Approaches for Probability Estimation Without Calibration Data.}},
author = {Eleonora Di Carluccio and Georgios Koliopanos and Francisco M Ojeda and Christian Weimar and Andreas Ziegler},
journal = {Statistics in medicine},
year = {2026},
month = oct,
doi = {10.1002/sim.70751},
url = {https://europepmc.org/article/MED/42894315},
note = {CC BY-NC},
}
RIS
TY - JOUR
TI - Calibrating Machine Learning Approaches for Probability Estimation Without Calibration Data.
AU - Eleonora Di Carluccio
AU - Georgios Koliopanos
AU - Francisco M Ojeda
AU - Christian Weimar
AU - Andreas Ziegler
PY - 2026
DA - 2026/10/01
JO - Statistics in medicine
DO - 10.1002/sim.70751
UR - https://europepmc.org/article/MED/42894315
AB - Statistical prediction models for binary outcomes are becoming increasingly popular. One significant challenge is calibrating these models to suit the characteristics of a target population that is structurally different from the original population. Calibration is especially challenging when there is no training data available from the target population. To address this problem, we propose a novel calibration method, SimCal, which uses synthetic data generated from the model development data in conjunction with marginal statistics from the calibration cohort. We show that expert judgment modeling (EJM) may be used for calibration if cross-sectional data from the target population are available comprising expert judgments about the potential outcome and the covariates. We describe three alternative calibration approaches when calibration data are lacking: similarity-binning averaging (SBA), adaptive calibration of predictions (ACP), and Elkan calibration. In a simulation study, we compare SBA, ACP, Elkan calibration, and SimCal. R code for applying these methods is provided from the re-analysis of data on coronary artery disease. We illustrate all 5 calibration approaches with a real data set for predicting functional outcome after stroke and all approaches but EJM in the re-analysis of the Cleveland Clinic data. None of the approaches performed convincingly well in all situations. SimCal performed well when model parameters were correctly specified. EJM failed on the stroke data. Further research is urgently required for calibration in the absence of calibration data.
SN - 0277-6715
SN - 1097-0258
ER -