Conformal Bayes under Continuous Label Shift: Sensitivity Analysis and the Limits of Exact Validity
Abstract
Conformal Bayes combines Bayesian posterior predictive scores with conformal calibration. Under continuous label shift, both the score and calibration weights depend on an unknown response density ratio. Existing methods typically estimate a single exponential tilt from pseudo-labels or predictive samples and plug it into calibration. We propose Joint Tilt-Sensitivity Conformal Bayes (JTS-CB), which performs sensitivity analysis over a prespecified family of tilts. Each tilt defines a coupled score-weight pair. The split procedure, JTS-SCB, solves joint min-max calibration by taking the largest tilt-specific threshold from weighted intercept-only quantile regression. JTS-Union retains individual thresholds, yielding prediction sets contained in those of JTS-SCB. Both use calibration-only weights by design, without a general coverage guarantee. We also study a candidate-weighted exact counterpart. Any nonzero linear exponential tilt makes its prediction set unbounded, whereas compact negative-quadratic families yield bounded exact unions under Gaussian scores. For the practical procedures, a localized correction with fallback gives coverage at least nominal minus a target tail-mass bound when the family contains the true tilt, but can substantially increase width. Experiments on controlled shifts of three regression datasets and on synthetic data show that plug-in predictive sampling can approach oracle coverage when the shift is well identified, but remains vulnerable to predictive bias and unstable tilt estimation. Sensitivity analysis can mitigate these failures in the tested settings, at the cost of wider prediction sets. Coupling scores and weights can yield narrower sets with higher coverage than varying weights alone, although this advantage is not uniform across shifts. Excluding the true tilt from the sensitivity family can still cause undercoverage.
Related
- Fully specified Bayes factors for hypothesis testing and sensitivity analysis in process tracing
- On Generalized Naive Bayes with Continuous Features
- Conformal Coverage of Time Series: Validity and Inference
- Improving the Sensitivity of Gravitational Wave Detection with Weighted Conformal Prediction
- On the validity limits of the parametrisation method for invariant manifolds: an assessment of practical criteria for vibrating systems
BibTeX
@misc{choi2026conformal,
title = {{Conformal Bayes under Continuous Label Shift: Sensitivity Analysis and the Limits of Exact Validity}},
author = {Seungjin Choi},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2609.27976},
eprint = {2609.27976},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2609.27976},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Conformal Bayes under Continuous Label Shift: Sensitivity Analysis and the Limits of Exact Validity
AU - Seungjin Choi
PY - 2026
DA - 2026/09/01
PB - arXiv
DO - 10.48550/arxiv.2609.27976
UR - https://arxiv.org/abs/2609.27976
AB - Conformal Bayes combines Bayesian posterior predictive scores with conformal calibration. Under continuous label shift, both the score and calibration weights depend on an unknown response density ratio. Existing methods typically estimate a single exponential tilt from pseudo-labels or predictive samples and plug it into calibration. We propose Joint Tilt-Sensitivity Conformal Bayes (JTS-CB), which performs sensitivity analysis over a prespecified family of tilts. Each tilt defines a coupled score-weight pair. The split procedure, JTS-SCB, solves joint min-max calibration by taking the largest tilt-specific threshold from weighted intercept-only quantile regression. JTS-Union retains individual thresholds, yielding prediction sets contained in those of JTS-SCB. Both use calibration-only weights by design, without a general coverage guarantee. We also study a candidate-weighted exact counterpart. Any nonzero linear exponential tilt makes its prediction set unbounded, whereas compact negative-quadratic families yield bounded exact unions under Gaussian scores. For the practical procedures, a localized correction with fallback gives coverage at least nominal minus a target tail-mass bound when the family contains the true tilt, but can substantially increase width. Experiments on controlled shifts of three regression datasets and on synthetic data show that plug-in predictive sampling can approach oracle coverage when the shift is well identified, but remains vulnerable to predictive bias and unstable tilt estimation. Sensitivity analysis can mitigate these failures in the tested settings, at the cost of wider prediction sets. Coupling scores and weights can yield narrower sets with higher coverage than varying weights alone, although this advantage is not uniform across shifts. Excluding the true tilt from the sensitivity family can still cause undercoverage.
ER -