Calibration Assessment for Nested Uncertainty Sets
Abstract
We present a Bayesian framework for assessing the calibration of nested uncertainty sets from simulation studies. Calibration is typically evaluated by repeatedly simulating data from a known model and comparing empirical coverage probabilities with their nominal values, an approach that often requires hundreds or thousands of independent model evaluations to obtain reliable estimates. Such computational demands can be prohibitive when the forward models are expensive to evaluate, for example in data assimilation using a large Earth-system model, or may not be applicable, for exmaple when the output of the simulation is a single or multiple uncertainty sets rather than a full distribution. We present a method that leverages the nested structure of uncertainty sets across nominal coverage levels to assess the calibration of these models. The method, which is based on a Dirichlet-Multinomial model, allows for testing both exact and region-based notions of model calibration in an efficient manner using Bayes factors. Unlike methods that require a full predictive distribution, this method works for procedures that directly produce prediction or confidence intervals. We demonstrate the method on a data assimilation problem containing model form error and show that it is capable of detecting miscalibration even when there are only a limited number of simulation studies available.
Related
- The Intersection of nested filled-in little Julia sets have zero area
- Convex Optimization with Nested Evolving Feasible Sets (CONES) under Time-Varying Loss Functions
- Nested MCMC for Uncertainty Propagation in Spatial Modeling of Temporal Long-Memory in Resting-State fMRI
- A Three-Way Testing Framework for Quantifying Epistemic Calibration Uncertainty in SBI
- ProbabilityBench: Model Evaluation and Calibration with Exact Ground-Truth Uncertainty
BibTeX
@misc{leduc2026calibration,
title = {{Calibration Assessment for Nested Uncertainty Sets}},
author = {Matthew LeDuc},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.04790},
eprint = {2610.04790},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.04790},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Calibration Assessment for Nested Uncertainty Sets
AU - Matthew LeDuc
PY - 2026
DA - 2026/10/03
PB - arXiv
DO - 10.48550/arxiv.2610.04790
UR - https://arxiv.org/abs/2610.04790
AB - We present a Bayesian framework for assessing the calibration of nested uncertainty sets from simulation studies. Calibration is typically evaluated by repeatedly simulating data from a known model and comparing empirical coverage probabilities with their nominal values, an approach that often requires hundreds or thousands of independent model evaluations to obtain reliable estimates. Such computational demands can be prohibitive when the forward models are expensive to evaluate, for example in data assimilation using a large Earth-system model, or may not be applicable, for exmaple when the output of the simulation is a single or multiple uncertainty sets rather than a full distribution. We present a method that leverages the nested structure of uncertainty sets across nominal coverage levels to assess the calibration of these models. The method, which is based on a Dirichlet-Multinomial model, allows for testing both exact and region-based notions of model calibration in an efficient manner using Bayes factors. Unlike methods that require a full predictive distribution, this method works for procedures that directly produce prediction or confidence intervals. We demonstrate the method on a data assimilation problem containing model form error and show that it is capable of detecting miscalibration even when there are only a limited number of simulation studies available.
ER -