Sequential Confidence Sets for Coverage-Constrained Conformal Model Selection
Abstract
Modern conformal forecasting systems often maintain several adaptive pipelines that differ in base forecasters, conformity scores, calibration windows, and update rules. Comparing them is difficult because coverage is a hard constraint, whereas efficiency should be optimized only among feasible pipelines. We formulate this problem as sequential inference for a stochastic constrained argmin. At each time, the target is the set of minimum-cost pipelines satisfying multiple prefix-average conditional miscoverage constraints. We introduce Coverage-Constrained Sequential Model Confidence Sets (CC-SMCS), which separate certifiably feasible, possibly feasible, and possibly constrained-optimal pipelines. Using simultaneous martingale confidence sequences, CC-SMCS projects a rectangular confidence region onto the constrained argmin and admits an exact closed-form rule. With probability at least 1−δ, it contains every constrained-optimal pipeline simultaneously over all times. This finite-sample guarantee requires no stationarity or mixing assumptions and remains valid under data-dependent stopping. We also establish an impossibility result for safe certification at the coverage boundary and extend the construction to delayed multi-horizon feedback and outcome-dependent efficiency objectives.
Related
- Selection-Safe Conformal Early Exit: Distribution-Free Coverage under Adaptive Depth
- Sequential Conformal Calibration under Recursive Selection
- FOCUS: Fixed-Confidence Online Causal Learning Using Sequential Adaptive Interventions
- Conformal Prediction and Conditional Coverage for Tabular Foundation Models
- Conformal Coverage of Time Series: Validity and Inference
BibTeX
@misc{li2026sequential,
title = {{Sequential Confidence Sets for Coverage-Constrained Conformal Model Selection}},
author = {Jing Li and Haibin Zhu},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2609.28522},
eprint = {2609.28522},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2609.28522},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Sequential Confidence Sets for Coverage-Constrained Conformal Model Selection
AU - Jing Li
AU - Haibin Zhu
PY - 2026
DA - 2026/09/22
PB - arXiv
DO - 10.48550/arxiv.2609.28522
UR - https://arxiv.org/abs/2609.28522
AB - Modern conformal forecasting systems often maintain several adaptive pipelines that differ in base forecasters, conformity scores, calibration windows, and update rules. Comparing them is difficult because coverage is a hard constraint, whereas efficiency should be optimized only among feasible pipelines. We formulate this problem as sequential inference for a stochastic constrained argmin. At each time, the target is the set of minimum-cost pipelines satisfying multiple prefix-average conditional miscoverage constraints. We introduce Coverage-Constrained Sequential Model Confidence Sets (CC-SMCS), which separate certifiably feasible, possibly feasible, and possibly constrained-optimal pipelines. Using simultaneous martingale confidence sequences, CC-SMCS projects a rectangular confidence region onto the constrained argmin and admits an exact closed-form rule. With probability at least 1−δ, it contains every constrained-optimal pipeline simultaneously over all times. This finite-sample guarantee requires no stationarity or mixing assumptions and remains valid under data-dependent stopping. We also establish an impossibility result for safe certification at the coverage boundary and extend the construction to delayed multi-horizon feedback and outcome-dependent efficiency objectives.
ER -