Testable and Actionable Calibration for Full Swap Regret
Abstract
AI generated predictions increasingly inform decision making in critical tasks, and therefore must be trustworthy. One widely used measure of trustworthiness is calibration, which requires that the predictions match the true frequencies and can be treated like real probabilities of a given outcome. However, defining calibration is subtle, and designing good measures of calibration error has been an active topic of recent research. The first goal is to find calibration measures that are actionable, meaning they can inform decision makers about their utility loss when predictions are treated as true probabilities, which is known as swap regret. The second goal is to find calibration measures that are testable, meaning that calibration error can be measured from a small sample of predictions and outcomes. Although these are very basic requirements, there is no existing calibration measure that fully satisfies both properties, and all existing measures relax actionability by bounding a weaker notion of swap regret, or relax testability by having suboptimal estimation error. We introduce a new calibration measure, Soft-Binned Calibration Decision Loss (SCDL), which we prove is fully actionable without weakening either requirement, and testable with nearly optimal error rate. In addition, SCDL satisfies other desired properties such as continuity and consistency. We also provide a set of experiments confirming that the theoretical advantages of SCDL compared to other measures lead to better performance in practice.
Related
- Resonance transformations for the (2,2p+1) minimal string via x−y swap: a proof of Artemev's conjecture
- A zero-one law for swap-invariant events in uniform spanning forests
- Variance and volatility swap valuation with nonlinear liquidity-feedback stochastic volatility and regime switching
- SWAP: A Scalable, Warm-Startable, Anytime Permutation Solver for Optimal Transport
- Modelling Skill-Based Queueing Systems using Pass-and-Swap Queues and Whittle Networks
BibTeX
@misc{bairaktari2026testable,
title = {{Testable and Actionable Calibration for Full Swap Regret}},
author = {Konstantina Bairaktari and Lunjia Hu and Huy L. Nguyen and Jonathan Ullman},
howpublished = {arXiv},
year = {2026},
month = nov,
doi = {10.48550/arxiv.2605.17749},
eprint = {2605.17749},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2605.17749},
}
RIS
TY - UNPB
TI - Testable and Actionable Calibration for Full Swap Regret
AU - Konstantina Bairaktari
AU - Lunjia Hu
AU - Huy L. Nguyen
AU - Jonathan Ullman
PY - 2026
DA - 2026/11/02
PB - arXiv
DO - 10.48550/arxiv.2605.17749
UR - https://arxiv.org/abs/2605.17749
AB - AI generated predictions increasingly inform decision making in critical tasks, and therefore must be trustworthy. One widely used measure of trustworthiness is calibration, which requires that the predictions match the true frequencies and can be treated like real probabilities of a given outcome. However, defining calibration is subtle, and designing good measures of calibration error has been an active topic of recent research. The first goal is to find calibration measures that are actionable, meaning they can inform decision makers about their utility loss when predictions are treated as true probabilities, which is known as swap regret. The second goal is to find calibration measures that are testable, meaning that calibration error can be measured from a small sample of predictions and outcomes. Although these are very basic requirements, there is no existing calibration measure that fully satisfies both properties, and all existing measures relax actionability by bounding a weaker notion of swap regret, or relax testability by having suboptimal estimation error. We introduce a new calibration measure, Soft-Binned Calibration Decision Loss (SCDL), which we prove is fully actionable without weakening either requirement, and testable with nearly optimal error rate. In addition, SCDL satisfies other desired properties such as continuity and consistency. We also provide a set of experiments confirming that the theoretical advantages of SCDL compared to other measures lead to better performance in practice.
ER -