Backward Conformal Prediction under Covariate Shift
Abstract
Backward conformal prediction (BCP) fixes the size of a prediction set and estimates its miscoverage, but assumes calibration and test data are exchangeable. We extend BCP to covariate shift through importance weighting, preserving its guarantees when the density ratio is known. We show that BCP's tight miscoverage report depends only on the relative weight assigned to missed calibration examples, explaining why a better domain classifier can sometimes produce a worse estimate. Motivated by the fact that BCP depends only on nonconformity scores, we propose estimating density ratios on sorted score vectors rather than high-dimensional inputs. We prove that this approach recovers the same report asymptotically under an interpretable condition and characterize the resulting bias otherwise. Experiments with simulated and benchmark distribution shifts show that the proposed method improves BCP's miscoverage estimates, while score-based weighting consistently yields more accurate estimates than embedding-based weighting.
Related
- Finite-Sample Optimal Conformal Prediction Sets under Covariate Shift
- Sharp training-conditional coverage for conformal prediction under covariate shift
- Sketched Calibration for Conformal Prediction under Covariate Shift
- Auditing Conformal Prediction for Language Models Under Inference-Time Model Shift
- Conformal Prediction under Exponential-Tilt Joint Shift
BibTeX
@misc{meghrazi2026backward,
title = {{Backward Conformal Prediction under Covariate Shift}},
author = {Amin Meghrazi and Yuanyuan Li},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.16048},
eprint = {2610.16048},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.16048},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Backward Conformal Prediction under Covariate Shift
AU - Amin Meghrazi
AU - Yuanyuan Li
PY - 2026
DA - 2026/10/12
PB - arXiv
DO - 10.48550/arxiv.2610.16048
UR - https://arxiv.org/abs/2610.16048
AB - Backward conformal prediction (BCP) fixes the size of a prediction set and estimates its miscoverage, but assumes calibration and test data are exchangeable. We extend BCP to covariate shift through importance weighting, preserving its guarantees when the density ratio is known. We show that BCP's tight miscoverage report depends only on the relative weight assigned to missed calibration examples, explaining why a better domain classifier can sometimes produce a worse estimate. Motivated by the fact that BCP depends only on nonconformity scores, we propose estimating density ratios on sorted score vectors rather than high-dimensional inputs. We prove that this approach recovers the same report asymptotically under an interpretable condition and characterize the resulting bias otherwise. Experiments with simulated and benchmark distribution shifts show that the proposed method improves BCP's miscoverage estimates, while score-based weighting consistently yields more accurate estimates than embedding-based weighting.
ER -