Careful Judge: Safe and Efficient Human-AI Collaborative Decision Making
Abstract
In human-AI collaborative decision making, human review can prevent unsafe AI decisions, but each human judgment is costly. Treating human intervention after AI abstention as a one-off fallback misses the opportunity to improve future AI decisions for greater automation, yet AI adaptively learning from selectively queried human feedback breaks safety guardrails calibrated for old models. We approach this challenge with CARE---calibrated adaptive rectification and escalation---an end-to-end pipeline that combines AI models and human reviewers to guarantee safe, human-aligned decisions, while continuously learning from human feedback to achieve greater automation with fewer human queries. CARE is principled, general, modular, and works with any black-box AI model. Our novel adaptive calibration module guarantees risk control at every time step for any rectification module. We further show how CARE improves query efficiency when the AI model is well trained and the human-AI misalignment has a clear structure. Experiments on four safety-critical real-world datasets spanning driving, language, and robotics demonstrate that CARE achieves human-aligned decisions while reducing human queries by 25-81% relative to baselines.
Related
- Classes of Aggregation Rules for Ethical Decision Making in Automated Systems
- Risk-Averse Decision Making with Multi-Level Reliability Guarantees
- Conformal Risk-Averse Decision Making with Optimized Certainty Equivalent Risk Control
- Value of Information in Dynamic Decision Making
- ElecMeter: An index for evaluating dissatisfaction of social decision-making
BibTeX
@misc{zhang2026careful,
title = {{Careful Judge: Safe and Efficient Human-AI Collaborative Decision Making}},
author = {Chenyu Zhang and Rachel Luo and Boyi Li and Anjali Parashar and Marco Pavone and Apoorva Sharma},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.09043},
eprint = {2610.09043},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.09043},
}
RIS
TY - UNPB
TI - Careful Judge: Safe and Efficient Human-AI Collaborative Decision Making
AU - Chenyu Zhang
AU - Rachel Luo
AU - Boyi Li
AU - Anjali Parashar
AU - Marco Pavone
AU - Apoorva Sharma
PY - 2026
DA - 2026/10/06
PB - arXiv
DO - 10.48550/arxiv.2610.09043
UR - https://arxiv.org/abs/2610.09043
AB - In human-AI collaborative decision making, human review can prevent unsafe AI decisions, but each human judgment is costly. Treating human intervention after AI abstention as a one-off fallback misses the opportunity to improve future AI decisions for greater automation, yet AI adaptively learning from selectively queried human feedback breaks safety guardrails calibrated for old models. We approach this challenge with CARE---calibrated adaptive rectification and escalation---an end-to-end pipeline that combines AI models and human reviewers to guarantee safe, human-aligned decisions, while continuously learning from human feedback to achieve greater automation with fewer human queries. CARE is principled, general, modular, and works with any black-box AI model. Our novel adaptive calibration module guarantees risk control at every time step for any rectification module. We further show how CARE improves query efficiency when the AI model is well trained and the human-AI misalignment has a clear structure. Experiments on four safety-critical real-world datasets spanning driving, language, and robotics demonstrate that CARE achieves human-aligned decisions while reducing human queries by 25-81% relative to baselines.
ER -