Valid Stopping in Adaptive Generator-Verifier Loops
Abstract
Numerous agentic workflows are based on a generator-verifier loop: a generator proposes candidates, a cheap verifier scores them, and the workflow terminates when a proposal is verified as good enough. The verifier typically proxies a more costly ground-truth oracle, and as the generator searches adaptively against it, false acceptances may accumulate. Proposals can pass the proxy but fail under the costlier ground-truth check. We study when to stop these loops while controlling the false discovery rate of the accepted proposals. Our construction introduces tools of independent interest in distribution-free statistical testing and conformal risk control, including analysis of e-values constructed through index betting and a novel conformal risk control procedure for non-monotone losses. We validate the approach in synthetic settings and on a protein-design benchmark.
Related
- How Many Posterior Samples? Calibrated Stopping for Adaptive Sensing
- A general recipe for data-adaptive anytime-valid envelopes for iterative algorithms
- Adaptive auditing of AI systems with anytime-valid guarantees
- Certified Corruption Budgets: Anytime-Valid Leaderboard Claims under Adaptive Rigging
- Adaptive stopping and forecasting of screening, rescreening and randomization in multicentre clinical trials
BibTeX
@misc{hegazy2026valid,
title = {{Valid Stopping in Adaptive Generator-Verifier Loops}},
author = {Mahmoud Hegazy and Michael I. Jordan and Aymeric Dieuleveut},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.06432},
eprint = {2610.06432},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.06432},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Valid Stopping in Adaptive Generator-Verifier Loops
AU - Mahmoud Hegazy
AU - Michael I. Jordan
AU - Aymeric Dieuleveut
PY - 2026
DA - 2026/10/05
PB - arXiv
DO - 10.48550/arxiv.2610.06432
UR - https://arxiv.org/abs/2610.06432
AB - Numerous agentic workflows are based on a generator-verifier loop: a generator proposes candidates, a cheap verifier scores them, and the workflow terminates when a proposal is verified as good enough. The verifier typically proxies a more costly ground-truth oracle, and as the generator searches adaptively against it, false acceptances may accumulate. Proposals can pass the proxy but fail under the costlier ground-truth check. We study when to stop these loops while controlling the false discovery rate of the accepted proposals. Our construction introduces tools of independent interest in distribution-free statistical testing and conformal risk control, including analysis of e-values constructed through index betting and a novel conformal risk control procedure for non-monotone losses. We validate the approach in synthetic settings and on a protein-design benchmark.
ER -