A Ranking Approach for Measuring Calibration
Abstract
When providing forecasted probabilities with a predictive model, the ideal model offers perfect calibration: the true probability of the outcome (i.e., the probability that Y=1) exactly matches the forecasted probability f(X). In practice, models inevitably exhibit calibration error, and it is therefore important to be able to measure this miscalibration to assess a model's reliability. The Expected Calibration Error (ECE) is the most widely used measure of miscalibration, but is known to be impossible to estimate the ECE with guaranteed accuracy in an assumption-free setting. In this work, we propose an alternative measure, the rankECE, that is based on comparing points with neighboring values of the predicted probability f(X). Our theoretical guarantees and empirical results establish that rankECE provides a better proxy for ECE as compared to binned approximations to ECE, which are the most commonly-used approximations in practice.
Related
- Entity Ranking with Covariates and Sparse Intrinsic Scores
- A hybrid QSPR-MCDM framework for ranking organic acids using topological indices
- An automated ranking platform for machine learning regression models for meat spoilage prediction using multi-spectral imaging and metabolic profiling
- Hidden Disparities in Ranking: Category-Aware Fairness via Distributional Stability
- Permutations from ranking independent random variables
BibTeX
@misc{chatterjee2026ranking,
title = {{A Ranking Approach for Measuring Calibration}},
author = {Anirban Chatterjee and Rina Foygel Barber},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2609.13100},
eprint = {2609.13100},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2609.13100},
}
RIS
TY - UNPB
TI - A Ranking Approach for Measuring Calibration
AU - Anirban Chatterjee
AU - Rina Foygel Barber
PY - 2026
DA - 2026/09/11
PB - arXiv
DO - 10.48550/arxiv.2609.13100
UR - https://arxiv.org/abs/2609.13100
AB - When providing forecasted probabilities with a predictive model, the ideal model offers perfect calibration: the true probability of the outcome (i.e., the probability that Y=1) exactly matches the forecasted probability f(X). In practice, models inevitably exhibit calibration error, and it is therefore important to be able to measure this miscalibration to assess a model's reliability. The Expected Calibration Error (ECE) is the most widely used measure of miscalibration, but is known to be impossible to estimate the ECE with guaranteed accuracy in an assumption-free setting. In this work, we propose an alternative measure, the rankECE, that is based on comparing points with neighboring values of the predicted probability f(X). Our theoretical guarantees and empirical results establish that rankECE provides a better proxy for ECE as compared to binned approximations to ECE, which are the most commonly-used approximations in practice.
ER -