Benchmarking Goodness-of-Fit and Calibration Algorithms for Logistic Regression Classifiers: A Large-Scale Simulation Study under Sparse Data
Abstract
Binary logistic regression is among the most widely used classification algorithms, yet a classifier is only trustworthy if its predicted probabilities are well calibrated. The classical checks -- the Pearson chi-square and deviance statistics -- break down precisely in the modern setting where predictors are continuous and the data are sparse (one covariate pattern per observation). Four decades of research have produced dozens of alternative goodness-of-fit and calibration algorithms, yet practitioners still default to the Hosmer-Lemeshow test because it ships with their software. This paper provides a unified taxonomy and a large-scale, reproducible simulation benchmark; more than twenty tests are implemented in the open-source R package ebrahim.gof. We evaluate them across five covariate distributions and four misspecification scenarios, with 10,000 replications each, measuring both Type I error and power. Several classical tests prove liberal, rejecting correct models far too often, while others have little power. A compact core -- McCullagh, Osius-Rojek, le Cessie-van Houwelingen, Stute-Zhu, and the GiViTI calibration test -- delivers the best balance of correct size and high power, and is consistently more powerful than the ubiquitous Hosmer-Lemeshow test. A low-birth-weight application reinforces the point: a model with omitted interactions slips past nearly every test, exposed only by pairing sensitive tests with a calibration (reliability) curve. We translate these findings into practical, evidence-based guidance for assessing logistic regression fit.
Versions
Related
- Reproducible benchmarking of four machine-learning classifiers on a large synthetic heart-disease competition dataset
- Benchmarking Bilevel Derivative-Free Optimization Algorithms
- On the bias distribution of pseudo-linear regression algorithms for recursive identification
- On the Calibration of Isotonic Distributional Regression
- Trace-Class Results for MCMC Algorithms for Student-t Regression Models
BibTeX
@misc{ebrahim2026benchmarking,
title = {{Benchmarking Goodness-of-Fit and Calibration Algorithms for Logistic Regression Classifiers: A Large-Scale Simulation Study under Sparse Data}},
author = {Ebrahim Khaled Ebrahim and Ahmed El-Kotory},
howpublished = {arXiv},
year = {2026},
month = jul,
doi = {10.48550/arxiv.2607.16344},
eprint = {2607.16344},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2607.16344},
}
RIS
TY - UNPB
TI - Benchmarking Goodness-of-Fit and Calibration Algorithms for Logistic Regression Classifiers: A Large-Scale Simulation Study under Sparse Data
AU - Ebrahim Khaled Ebrahim
AU - Ahmed El-Kotory
PY - 2026
DA - 2026/07/16
PB - arXiv
DO - 10.48550/arxiv.2607.16344
UR - https://arxiv.org/abs/2607.16344
AB - Binary logistic regression is among the most widely used classification algorithms, yet a classifier is only trustworthy if its predicted probabilities are well calibrated. The classical checks -- the Pearson chi-square and deviance statistics -- break down precisely in the modern setting where predictors are continuous and the data are sparse (one covariate pattern per observation). Four decades of research have produced dozens of alternative goodness-of-fit and calibration algorithms, yet practitioners still default to the Hosmer-Lemeshow test because it ships with their software. This paper provides a unified taxonomy and a large-scale, reproducible simulation benchmark; more than twenty tests are implemented in the open-source R package ebrahim.gof. We evaluate them across five covariate distributions and four misspecification scenarios, with 10,000 replications each, measuring both Type I error and power. Several classical tests prove liberal, rejecting correct models far too often, while others have little power. A compact core -- McCullagh, Osius-Rojek, le Cessie-van Houwelingen, Stute-Zhu, and the GiViTI calibration test -- delivers the best balance of correct size and high power, and is consistently more powerful than the ubiquitous Hosmer-Lemeshow test. A low-birth-weight application reinforces the point: a model with omitted interactions slips past nearly every test, exposed only by pairing sensitive tests with a calibration (reliability) curve. We translate these findings into practical, evidence-based guidance for assessing logistic regression fit.
ER -