Calibrated Predictive Distributions from Sample-Based Generators
Abstract
Conditional generative models, including diffusion models and ensemble forecasters, often produce predictive samples without a tractable likelihood representation. Such sample-based predictive distributions can be systematically biased and poorly calibrated. We propose bias-corrected conformal probability integral transform (PIT) calibration, a split-sample post-processing framework that outputs a calibrated predictive distribution rather than a single fixed-level prediction interval. The method first estimates an affine location-scale correction on a held-out bias split, then calibrates randomized PIT values using a conformal calibrator. The resulting predictive law is represented as a weighted empirical distribution on the generator order statistics, enabling the direct computation of threshold-coherent exceedance probabilities, arbitrary quantiles, highest-density intervals, expected tail losses, and calibrated resamples. In contrast, standard conformal prediction primarily provides fixed-level prediction sets or threshold decisions and does not directly estimate predictive probabilities or high-density regions. We establish finite-sample calibration in probability under exchangeability and show how an optional split-conformal wrapper based on a PIT-centrality score gives nested prediction intervals with finite-sample marginal coverage at user-specified levels. Simulation studies with controlled misspecification and a WeatherBench-2 precipitation-forecasting application demonstrate substantial improvements in probabilistic calibration and downstream distributional summaries relative to uncalibrated sample-based forecasts and interval-only conformal baselines.
Related
- SCOPE Before You Sample: Conformal Scores Guide Posterior Exploration for Fast and Calibrated Bayesian Predictive Inference
- Locally calibrated and mesh-free inference for spatial point distributions: closed-form null, contamination law, and detectability threshold
- Bagged Martingale Posteriors: Calibrated Uncertainty Quantification for Predictive Resampling
- Calibrated Multiaccuracy: Deterministic and Randomized Sample Complexity
- How to Run Statistics over LLM Judges and Trust the Results: Calibrated Inference for Small-Sample AI Evaluation with evalstats
BibTeX
@misc{wang2026calibrated,
title = {{Calibrated Predictive Distributions from Sample-Based Generators}},
author = {Wen-Ting Wang and ShengLi Tzeng and Yu-Ting Fan and Hsin-Cheng Huang},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2609.19035},
eprint = {2609.19035},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2609.19035},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Calibrated Predictive Distributions from Sample-Based Generators
AU - Wen-Ting Wang
AU - ShengLi Tzeng
AU - Yu-Ting Fan
AU - Hsin-Cheng Huang
PY - 2026
DA - 2026/09/01
PB - arXiv
DO - 10.48550/arxiv.2609.19035
UR - https://arxiv.org/abs/2609.19035
AB - Conditional generative models, including diffusion models and ensemble forecasters, often produce predictive samples without a tractable likelihood representation. Such sample-based predictive distributions can be systematically biased and poorly calibrated. We propose bias-corrected conformal probability integral transform (PIT) calibration, a split-sample post-processing framework that outputs a calibrated predictive distribution rather than a single fixed-level prediction interval. The method first estimates an affine location-scale correction on a held-out bias split, then calibrates randomized PIT values using a conformal calibrator. The resulting predictive law is represented as a weighted empirical distribution on the generator order statistics, enabling the direct computation of threshold-coherent exceedance probabilities, arbitrary quantiles, highest-density intervals, expected tail losses, and calibrated resamples. In contrast, standard conformal prediction primarily provides fixed-level prediction sets or threshold decisions and does not directly estimate predictive probabilities or high-density regions. We establish finite-sample calibration in probability under exchangeability and show how an optional split-conformal wrapper based on a PIT-centrality score gives nested prediction intervals with finite-sample marginal coverage at user-specified levels. Simulation studies with controlled misspecification and a WeatherBench-2 precipitation-forecasting application demonstrate substantial improvements in probabilistic calibration and downstream distributional summaries relative to uncalibrated sample-based forecasts and interval-only conformal baselines.
ER -