Confidence regions for SAFE AI compliance scores
Abstract
Artificial Intelligence trustworthiness scores are moving from research dashboards into compliance claims and procurement decisions. A compliance claim is a statistical decision based on the comparison between an estimated compliance metric and a set threshold. The recently proposed integrated SAFE AI metrics are based on three basic rank-graduation measures of accuracy, explainability and robustness, expressed on a common footing. The metrics are integrated into a single compliance score, so far reported as a point value. In this paper we propose to add an uncertainty layer to the SAFE AI compliance score. All component metrics are estimated on a shared test sample, so their errors covary. We estimate the full covariance of the component vector with a paired bootstrap, and quantify departures from the source-wise independence baseline suggested by the integrated-metrics framework's variability decomposition. A closed-form identity splits such departure into an uncertainty-weighted effective dimension and an exposure-weighted data-driven correlation, with both depending on the aggregator and the estimated uncertainty of the component metrics. The application of our proposal to both real and simulated data shows that the assumption of independence can either narrow or widen confidence bands, while the direction and magnitude of the effect depend on the aggregator type, machine learning model, and perturbation family. The resulting confidence region supports compliance decisions, which can be documented in a formal uncertainty certificate.
Related
- Non-Asymptotic Confidence Regions for Separable Nonlinear Models
- From matrix inversion to constraints: provably tighter confidence regions for importance weights in label shift
- One-Shot Private Confidence Regions via Resampling
- Active learning strategy for excursion-set confidence regions of functional simulator outputs
- Overlap-free multi-material topology optimization for minimum compliance in two and three dimensions by level-set-based negative-mapping interpolation
BibTeX
@misc{sokolov2026confidence,
title = {{Confidence regions for SAFE AI compliance scores}},
author = {Anton Sokolov and Paolo Giudici and Vasily Kolesnikov},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2609.29588},
eprint = {2609.29588},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2609.29588},
}
RIS
TY - UNPB
TI - Confidence regions for SAFE AI compliance scores
AU - Anton Sokolov
AU - Paolo Giudici
AU - Vasily Kolesnikov
PY - 2026
DA - 2026/09/01
PB - arXiv
DO - 10.48550/arxiv.2609.29588
UR - https://arxiv.org/abs/2609.29588
AB - Artificial Intelligence trustworthiness scores are moving from research dashboards into compliance claims and procurement decisions. A compliance claim is a statistical decision based on the comparison between an estimated compliance metric and a set threshold. The recently proposed integrated SAFE AI metrics are based on three basic rank-graduation measures of accuracy, explainability and robustness, expressed on a common footing. The metrics are integrated into a single compliance score, so far reported as a point value. In this paper we propose to add an uncertainty layer to the SAFE AI compliance score. All component metrics are estimated on a shared test sample, so their errors covary. We estimate the full covariance of the component vector with a paired bootstrap, and quantify departures from the source-wise independence baseline suggested by the integrated-metrics framework's variability decomposition. A closed-form identity splits such departure into an uncertainty-weighted effective dimension and an exposure-weighted data-driven correlation, with both depending on the aggregator and the estimated uncertainty of the component metrics. The application of our proposal to both real and simulated data shows that the assumption of independence can either narrow or widen confidence bands, while the direction and magnitude of the effect depend on the aggregator type, machine learning model, and perturbation family. The resulting confidence region supports compliance decisions, which can be documented in a formal uncertainty certificate.
ER -