A Three-Way Testing Framework for Quantifying Epistemic Calibration Uncertainty in SBI
Abstract
Current experimental scientists increasingly rely on simulation-based inference (SBI) to invert complex models with intractable likelihoods. A primary goal in these settings is to obtain credible regions with valid coverage. While recent model-agnostic conformal calibration methods have succeeded in constructing credible sets with prescribed local Bayesian coverage, their approximate nature introduces inherent epistemic uncertainty in the calibration process. In this work, we propose a novel tool for diagnosing calibration uncertainty. Our approach is based on a simple three-way hypothesis testing procedure. We demonstrate how this tool can be used to effectively assess necessary simulation budgets for calibration sets and analyze the epistemic uncertainty associated with cutoff estimation.
Related
- Hybrid Uncertainty Sensitivity Analysis Based on the HSIC for High-Dimensional Responses with Aleatory--Epistemic Separation
- Post-hoc Epistemic Uncertainty for Vision-Language-Action Policies
- ProbabilityBench: Model Evaluation and Calibration with Exact Ground-Truth Uncertainty
- Finite-horizon maintenance under geometric-process models: Quantifying model and decision uncertainty
- Calibration Assessment for Nested Uncertainty Sets
BibTeX
@misc{cabezas2026threeway,
title = {{A Three-Way Testing Framework for Quantifying Epistemic Calibration Uncertainty in SBI}},
author = {Luben M. C. Cabezas and Pedro L. C. Rodrigues and Rafael Izbicki},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2609.24419},
eprint = {2609.24419},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2609.24419},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - A Three-Way Testing Framework for Quantifying Epistemic Calibration Uncertainty in SBI
AU - Luben M. C. Cabezas
AU - Pedro L. C. Rodrigues
AU - Rafael Izbicki
PY - 2026
DA - 2026/09/21
PB - arXiv
DO - 10.48550/arxiv.2609.24419
UR - https://arxiv.org/abs/2609.24419
AB - Current experimental scientists increasingly rely on simulation-based inference (SBI) to invert complex models with intractable likelihoods. A primary goal in these settings is to obtain credible regions with valid coverage. While recent model-agnostic conformal calibration methods have succeeded in constructing credible sets with prescribed local Bayesian coverage, their approximate nature introduces inherent epistemic uncertainty in the calibration process. In this work, we propose a novel tool for diagnosing calibration uncertainty. Our approach is based on a simple three-way hypothesis testing procedure. We demonstrate how this tool can be used to effectively assess necessary simulation budgets for calibration sets and analyze the epistemic uncertainty associated with cutoff estimation.
ER -