Reliable AI-Assisted Decisions via Selective Auditing
Abstract
We consider online monitoring of an AI system that produces a sequence of decisions. At each round, the system outputs a proposed decision together with a confidence score estimating the trustworthiness of the decision. The decision and confidence score need not be produced by the same model: for example, one agent may propose a decision, while a separate verifier agent evaluates it and outputs a correctness or safety score. Based on this information, we must decide whether to accept the AI decision automatically or consult an expert, such as a human auditor, who reviews the proposal and either accepts or corrects it. We formulate this task as an online constrained optimization problem in which the goal is to minimize the frequency of expert auditing while guaranteeing that the fraction of AI errors that go unaudited stays below a user-defined threshold. We derive a family of algorithms with provable error-control guarantees with no assumptions, including a method that is optimally efficient: it asymptotically attains the best achievable auditing rate for both adversarial and i.i.d. data streams. Experiments on screening harmful requests to large language models, qualitative text annotation, and data labeling show that our methods achieve valid error control in practice while substantially reducing the need for human auditing compared to baselines.
Related
- Where the LLM Ends and Reliable Decisions Begin
- Abstention Spends Its Budget on Local Label Variance More Than on Novelty: Auditing Conformal Calibration and Selective Prediction under Molecular Scaffold Shift
- Empirical Auditing of Edge-Private Graph Generators
- PDECheck: A Diagnostic Software Suite for Auditing Learned PDE Simulators
- Integrating Network Psychometrics and LLMs: The Ising-Embeddings-Model applied to Reliability Auditing
BibTeX
@misc{gibbs2026reliable,
title = {{Reliable AI-Assisted Decisions via Selective Auditing}},
author = {Isaac Gibbs and Elise L. Han and Ryan J. Tibshirani and Tijana Zrnic},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.31662},
eprint = {2610.31662},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.31662},
}
RIS
TY - UNPB
TI - Reliable AI-Assisted Decisions via Selective Auditing
AU - Isaac Gibbs
AU - Elise L. Han
AU - Ryan J. Tibshirani
AU - Tijana Zrnic
PY - 2026
DA - 2026/10/22
PB - arXiv
DO - 10.48550/arxiv.2610.31662
UR - https://arxiv.org/abs/2610.31662
AB - We consider online monitoring of an AI system that produces a sequence of decisions. At each round, the system outputs a proposed decision together with a confidence score estimating the trustworthiness of the decision. The decision and confidence score need not be produced by the same model: for example, one agent may propose a decision, while a separate verifier agent evaluates it and outputs a correctness or safety score. Based on this information, we must decide whether to accept the AI decision automatically or consult an expert, such as a human auditor, who reviews the proposal and either accepts or corrects it. We formulate this task as an online constrained optimization problem in which the goal is to minimize the frequency of expert auditing while guaranteeing that the fraction of AI errors that go unaudited stays below a user-defined threshold. We derive a family of algorithms with provable error-control guarantees with no assumptions, including a method that is optimally efficient: it asymptotically attains the best achievable auditing rate for both adversarial and i.i.d. data streams. Experiments on screening harmful requests to large language models, qualitative text annotation, and data labeling show that our methods achieve valid error control in practice while substantially reducing the need for human auditing compared to baselines.
ER -