One Goodness-of-Fit Test Is Not Enough: Error-Controlled Localization of Misfit in Logistic Risk Models
Abstract
A goodness-of-fit test tells whether a logistic risk model fits, but the analyst also needs to know what is wrong: an overall risk that is too high is repaired by updating the intercept, risks that are too extreme by recalibration, a curved relation between score and risk by a flexible recalibration, and misfit within groups of patients only by revising the model. We split any misfit into these four parts, which are orthogonal and match the calibration hierarchy. Each part gets a group of score tests that sees that part only, and the failing parts are named by closed testing. In external validation, in large samples and for small departures, a part is wrongly named with probability at most the chosen level, and the highest part named points to the lightest update that removes the misfit. Checking a model on its development data never sees an error in the overall risk or the calibration slope, and with binary or skewed covariates it can mistake misfit within patient groups for a curved link. External validation sees all four parts, for any prediction model. In simulations it kept its error rate when the model was right and for small departures, and usually pointed to the right update. On intensive-care data its verdicts for one model differed in-sample, on similar patients and after transport.
Related
- Shrinkage invalidates the Hosmer-Lemeshow test: goodness of fit for penalized logistic regression, with an application to glaucoma diagnosis
- Benchmarking Goodness-of-Fit and Calibration Algorithms for Logistic Regression Classifiers: A Large-Scale Simulation Study under Sparse Data
- [DeepGOF] Where Does a Logistic Risk Model Fail? An Audited Neural Goodness-of-Fit Test for Model Development and External Validation
- Data generation for the logistic regression model under controlled conditions
- Data-Matrix-Based Overall Goodness-of-Fit Testing for Confirmatory Factor Analysis: In-Sample Prediction Error for Observed Variables
BibTeX
@misc{ebrahim2026goodnessoffit,
title = {{One Goodness-of-Fit Test Is Not Enough: Error-Controlled Localization of Misfit in Logistic Risk Models}},
author = {Ebrahim Khaled Ebrahim and Ahmed El-Kotory and Osama Abd El-Aziz Hussein},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.15649},
eprint = {2610.15649},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.15649},
}
RIS
TY - UNPB
TI - One Goodness-of-Fit Test Is Not Enough: Error-Controlled Localization of Misfit in Logistic Risk Models
AU - Ebrahim Khaled Ebrahim
AU - Ahmed El-Kotory
AU - Osama Abd El-Aziz Hussein
PY - 2026
DA - 2026/10/11
PB - arXiv
DO - 10.48550/arxiv.2610.15649
UR - https://arxiv.org/abs/2610.15649
AB - A goodness-of-fit test tells whether a logistic risk model fits, but the analyst also needs to know what is wrong: an overall risk that is too high is repaired by updating the intercept, risks that are too extreme by recalibration, a curved relation between score and risk by a flexible recalibration, and misfit within groups of patients only by revising the model. We split any misfit into these four parts, which are orthogonal and match the calibration hierarchy. Each part gets a group of score tests that sees that part only, and the failing parts are named by closed testing. In external validation, in large samples and for small departures, a part is wrongly named with probability at most the chosen level, and the highest part named points to the lightest update that removes the misfit. Checking a model on its development data never sees an error in the overall risk or the calibration slope, and with binary or skewed covariates it can mistake misfit within patient groups for a curved link. External validation sees all four parts, for any prediction model. In simulations it kept its error rate when the model was right and for small departures, and usually pointed to the right update. On intensive-care data its verdicts for one model differed in-sample, on similar patients and after transport.
ER -