Lifetime-valid decisions by continually evolving AI analysts
Abstract
AI agents increasingly act as analysts whose decision can often be cast as hypothesis tests. Classical tests control the Type I error rate, the probability of rejecting a true null hypothesis, for fixed procedures. However, AI analysts introduce internal randomness and can evolve continuously over time. As AI analysts evolve, the risk of approving versions with Type I error rates above the target can accumulate, and adaptive use of evaluation feedback can lead to overfitting. We introduce continual validation, which evaluates each proposed AI analyst version before deployment using simulated data under the null hypothesis, with scenarios that can be randomized and expanded over time. We prove that, regardless of how updates are chosen or how many are proposed, the lifetime probability of ever deploying a version whose Type I error rate exceeds the allowed level remains bounded. This guarantees that every subsequent decision controls the Type I error rate at the target level across these scenarios. At the same time, updates with sufficiently low Type I error rates pass validation with high probability, allowing the analyst to continuously improve its performance without sacrificing rigor. In two examples involving evolving AI analysts for bioequivalence studies and product A/B tests, Type I error rates reached 0.098 and 0.183 without continual validation, compared with at most 0.045 and 0.044 with it, while retaining 82% and 96% of the power from optimally selected valid updates.
Related
- An Efficient Monte Carlo Method for Valid Prior-Free Possibilistic Statistical Inference
- Anytime-Valid Inference in Linear Models with Applications to Regression-Adjusted Causal Inference
- Valid Data‐Driven Subgroup Selection With Uniform Guarantees in Clinical Trials
- Valid Stopping in Adaptive Generator-Verifier Loops
- Asymptotic Anytime-Valid Quantile Inference under Local Differential Privacy
BibTeX
@misc{sun2026lifetimevalid,
title = {{Lifetime-valid decisions by continually evolving AI analysts}},
author = {Xiaojing Sun and Yuhan Zeng and Xiao Wang},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.30562},
eprint = {2610.30562},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.30562},
}
RIS
TY - UNPB
TI - Lifetime-valid decisions by continually evolving AI analysts
AU - Xiaojing Sun
AU - Yuhan Zeng
AU - Xiao Wang
PY - 2026
DA - 2026/10/22
PB - arXiv
DO - 10.48550/arxiv.2610.30562
UR - https://arxiv.org/abs/2610.30562
AB - AI agents increasingly act as analysts whose decision can often be cast as hypothesis tests. Classical tests control the Type I error rate, the probability of rejecting a true null hypothesis, for fixed procedures. However, AI analysts introduce internal randomness and can evolve continuously over time. As AI analysts evolve, the risk of approving versions with Type I error rates above the target can accumulate, and adaptive use of evaluation feedback can lead to overfitting. We introduce continual validation, which evaluates each proposed AI analyst version before deployment using simulated data under the null hypothesis, with scenarios that can be randomized and expanded over time. We prove that, regardless of how updates are chosen or how many are proposed, the lifetime probability of ever deploying a version whose Type I error rate exceeds the allowed level remains bounded. This guarantees that every subsequent decision controls the Type I error rate at the target level across these scenarios. At the same time, updates with sufficiently low Type I error rates pass validation with high probability, allowing the analyst to continuously improve its performance without sacrificing rigor. In two examples involving evolving AI analysts for bioequivalence studies and product A/B tests, Type I error rates reached 0.098 and 0.183 without continual validation, compared with at most 0.045 and 0.044 with it, while retaining 82% and 96% of the power from optimally selected valid updates.
ER -