Bet on Features: Anytime-Valid and Feature-Aware Auditing of Conditional Quantile Forecasters
Abstract
Black-box conditional quantile forecasts are widely used for sequential decisions under asymmetric costs. Once deployed, such forecasters must be monitored continuously as data streams drift and regimes change; this invalidates standard, fixed-horizon backtests for calibration. Further, existing backtests do not take into account that the notion of calibration is, in fact, information-dependent: forecasts can look calibrated to an auditor with coarse information while being miscalibrated to an auditor with richer information. We develop a distribution-free and game-theoretic testing framework for continuously auditing such forecasters with dependent, potentially nonstationary observations. We first formalize notions of conditional quantile calibration relative to the contextual features available to the auditor, showing how coarser information can limit detection. We then identify detectable alternative classes and, for contextual bets linear in the features, derive finite-time detection guarantees without an i.i.d. assumption. The resulting evidence processes provide interpretable, "feature-aware" evidence for miscalibration. We empirically validate these methods on simulated and real data, finding that a popular time series forecaster (Chronos-2) is highly miscalibrated w.r.t. multiple relevant features.
Related
- Asymptotic Anytime-Valid Quantile Inference under Local Differential Privacy
- CAQ: Clustered Asymptotically Anytime-Valid Quantile Treatment Effects for Online Experiments
- Adaptive auditing of AI systems with anytime-valid guarantees
- Multiscale Sequential Auditing of Quantile Regression with Point Masses
- Global tree forecasters collapse at the hierarchical aggregate: a five-panel failure characterization
BibTeX
@misc{antonov2026features,
title = {{Bet on Features: Anytime-Valid and Feature-Aware Auditing of Conditional Quantile Forecasters}},
author = {Ivane Antonov and Sohom Mukherjee and Richard Pibernik and Yo Joong Choe},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2607.11653},
eprint = {2607.11653},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2607.11653},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Bet on Features: Anytime-Valid and Feature-Aware Auditing of Conditional Quantile Forecasters
AU - Ivane Antonov
AU - Sohom Mukherjee
AU - Richard Pibernik
AU - Yo Joong Choe
PY - 2026
DA - 2026/10/28
PB - arXiv
DO - 10.48550/arxiv.2607.11653
UR - https://arxiv.org/abs/2607.11653
AB - Black-box conditional quantile forecasts are widely used for sequential decisions under asymmetric costs. Once deployed, such forecasters must be monitored continuously as data streams drift and regimes change; this invalidates standard, fixed-horizon backtests for calibration. Further, existing backtests do not take into account that the notion of calibration is, in fact, information-dependent: forecasts can look calibrated to an auditor with coarse information while being miscalibrated to an auditor with richer information. We develop a distribution-free and game-theoretic testing framework for continuously auditing such forecasters with dependent, potentially nonstationary observations. We first formalize notions of conditional quantile calibration relative to the contextual features available to the auditor, showing how coarser information can limit detection. We then identify detectable alternative classes and, for contextual bets linear in the features, derive finite-time detection guarantees without an i.i.d. assumption. The resulting evidence processes provide interpretable, "feature-aware" evidence for miscalibration. We empirically validate these methods on simulated and real data, finding that a popular time series forecaster (Chronos-2) is highly miscalibrated w.r.t. multiple relevant features.
ER -