CalArena: A Large-Scale Post-Hoc Calibration Benchmark
Abstract
Reliable probability estimates are critical in many machine learning applications, yet modern classifiers are often poorly calibrated. Post-hoc calibration provides a simple and widely used solution, but the large number of proposed methods, combined with small-scale and inconsistent evaluations, makes it difficult to determine which approaches are truly effective in practice. We introduce a large-scale, standardized benchmark for post-hoc calibration, covering nearly 2000 experiments across tabular and computer vision tasks, including binary, multiclass, and large-scale classification settings. Our benchmark aggregates predictions from a diverse set of classical models, modern deep learning architectures, and foundation models, and provides unified, reproducible implementations of dozens of calibration methods within a common evaluation framework. We argue that Post-Hoc Improvement (PHI) in proper scoring rules offers a principled alternative to traditional calibration error estimators for comparing post-hoc methods, capturing both calibration quality and potential degradation to the model's predictive performance. Using this framework, we conduct the most comprehensive empirical study of post-hoc calibration to date. Our results reveal consistent patterns across domains: smooth calibration functions outperform binning-based approaches, dedicated multiclass methods are essential in high-dimensional settings, and generic machine learning models are not competitive without calibration-specific design. To facilitate future research, we release all data, code, and evaluation tools, providing a plug-and-play benchmark for developing and comparing calibration methods.
Related
- Benchmarking Goodness-of-Fit and Calibration Algorithms for Logistic Regression Classifiers: A Large-Scale Simulation Study under Sparse Data
- TypedBench: A Benchmark for Calibration, Framing Sensitivity, and Cost in System One Decision Models
- Quantifying Portfolio Demutualization: A Benchmark-Relative Pooling--Profiling Scale
- Can Large Language Model-Generated Responses Support Assessment Development? A Human-Calibrated Rasch Benchmark
- Comparative Analysis of Large-Scale SVM Solvers: Convergence, Resource Consumption, and Energy Footprint
BibTeX
@misc{berta2026calarena,
title = {{CalArena: A Large-Scale Post-Hoc Calibration Benchmark}},
author = {Eugène Berta and David Holzmüller and Francis Bach and Michael I. Jordan},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2605.30188},
eprint = {2605.30188},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2605.30188},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - CalArena: A Large-Scale Post-Hoc Calibration Benchmark
AU - Eugène Berta
AU - David Holzmüller
AU - Francis Bach
AU - Michael I. Jordan
PY - 2026
DA - 2026/10/27
PB - arXiv
DO - 10.48550/arxiv.2605.30188
UR - https://arxiv.org/abs/2605.30188
AB - Reliable probability estimates are critical in many machine learning applications, yet modern classifiers are often poorly calibrated. Post-hoc calibration provides a simple and widely used solution, but the large number of proposed methods, combined with small-scale and inconsistent evaluations, makes it difficult to determine which approaches are truly effective in practice. We introduce a large-scale, standardized benchmark for post-hoc calibration, covering nearly 2000 experiments across tabular and computer vision tasks, including binary, multiclass, and large-scale classification settings. Our benchmark aggregates predictions from a diverse set of classical models, modern deep learning architectures, and foundation models, and provides unified, reproducible implementations of dozens of calibration methods within a common evaluation framework. We argue that Post-Hoc Improvement (PHI) in proper scoring rules offers a principled alternative to traditional calibration error estimators for comparing post-hoc methods, capturing both calibration quality and potential degradation to the model's predictive performance. Using this framework, we conduct the most comprehensive empirical study of post-hoc calibration to date. Our results reveal consistent patterns across domains: smooth calibration functions outperform binning-based approaches, dedicated multiclass methods are essential in high-dimensional settings, and generic machine learning models are not competitive without calibration-specific design. To facilitate future research, we release all data, code, and evaluation tools, providing a plug-and-play benchmark for developing and comparing calibration methods.
ER -