Conformal Policy Selection: When to Deploy AI Decisions for Targeted Intervention
Abstract
Artificial-intelligence (AI) policies are increasingly used to make consequential decisions in medicine, finance, education, and public services, yet deploying them responsibly requires knowing not only whether a policy helps on average, but for whom. Existing tools for off-policy evaluation and safe policy learning provide population-level guarantees, but some individuals may experience adverse treatment effects. We recast targeted deployment as a multiple-testing problem: for each candidate unit, test whether the outcome under the AI policy would exceed a user-specified threshold, and control the false discovery rate (FDR) among the selected set. Our method, Conformal Policy Selection, combines weighted conformal p-values that account for the distributional shift from behavior to target policy with a two-step selection-and-pruning procedure. We prove finite-sample FDR control and establish a doubly robust asymptotic guarantee, where FDR remains controlled if either the outcome model or the propensity model is correctly specified. Synthetic experiments verify these guarantees across four misspecification regimes, and a semi-synthetic study built on real electronic-health-record covariates illustrates how the same FDR control can act as a deployment safety gate, bounding the harmful treatments that a globally trained policy would otherwise administer to a high-risk subpopulation where it misgeneralizes. The framework offers a principled method for safe AI deployment in high-stakes applications.
Related
- Calibrated for Which Policy? Conformal Uncertainty for Fixed World Models Under Intervention
- Recertifying decisions via e-closure: uniform improvements of conformal risk control and randomized e-BH
- ACS: An interactive framework for conformal selection
- Conformal Policy Learning with Distribution-Free Safety Guarantees
- Sequential Confidence Sets for Coverage-Constrained Conformal Model Selection
BibTeX
@misc{cao2026conformal,
title = {{Conformal Policy Selection: When to Deploy AI Decisions for Targeted Intervention}},
author = {Junyu Cao and Ruijiang Gao and Maytal Saar-Tsechansky and Mingzhang Yin},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.22930},
eprint = {2610.22930},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.22930},
note = {CC BY-NC-ND 4.0},
}
RIS
TY - UNPB
TI - Conformal Policy Selection: When to Deploy AI Decisions for Targeted Intervention
AU - Junyu Cao
AU - Ruijiang Gao
AU - Maytal Saar-Tsechansky
AU - Mingzhang Yin
PY - 2026
DA - 2026/10/16
PB - arXiv
DO - 10.48550/arxiv.2610.22930
UR - https://arxiv.org/abs/2610.22930
AB - Artificial-intelligence (AI) policies are increasingly used to make consequential decisions in medicine, finance, education, and public services, yet deploying them responsibly requires knowing not only whether a policy helps on average, but for whom. Existing tools for off-policy evaluation and safe policy learning provide population-level guarantees, but some individuals may experience adverse treatment effects. We recast targeted deployment as a multiple-testing problem: for each candidate unit, test whether the outcome under the AI policy would exceed a user-specified threshold, and control the false discovery rate (FDR) among the selected set. Our method, Conformal Policy Selection, combines weighted conformal p-values that account for the distributional shift from behavior to target policy with a two-step selection-and-pruning procedure. We prove finite-sample FDR control and establish a doubly robust asymptotic guarantee, where FDR remains controlled if either the outcome model or the propensity model is correctly specified. Synthetic experiments verify these guarantees across four misspecification regimes, and a semi-synthetic study built on real electronic-health-record covariates illustrates how the same FDR control can act as a deployment safety gate, bounding the harmful treatments that a globally trained policy would otherwise administer to a high-risk subpopulation where it misgeneralizes. The framework offers a principled method for safe AI deployment in high-stakes applications.
ER -