Selection-Aware Stress Testing for Interactive Agents
Abstract
Agent evaluations often use one benchmark to choose a workflow and then search for task types where its advantage weakens, so both conclusions are selected from the same data. We introduce Selection-Aware Semantic Stress Testing (\SASST{}), which learns a task reweighting from pre-execution features on discovery tasks and evaluates the same paired comparison on separate confirmation tasks. The protocol checks support and stability, uses joint bounds for all planned claims, and can return no claim. We prove conditional asymptotic validity under stated cluster assumptions. A forty-cluster audit finds Gaussian undercoverage and conservative Bonferroni t bounds. In one 480-episode τ-bench study, a $3.75$ point discovery gain vanished on confirmation. A second-model study likewise confirmed neither a workflow benefit nor a stable stress rule.
Related
- ACS: An interactive framework for conformal selection
- Causal Retention in Interactive Agents: Interface Factorization and Selective Adaptation
- Hypothesis Testing for Nonlinear and Interactive Effects Using Additive Gaussian Processes
- GTDD: Generative Test-Driven Development for AI Coding Agents with Adversarial Testing
- Think Short, Defer Smart, Act, and Repeat: Calibrated Reasoning and Uncertainty-Aware Deferral for Edge LLM Agents
BibTeX
@misc{xu2026selectionaware,
title = {{Selection-Aware Stress Testing for Interactive Agents}},
author = {Yang Xu and Chenang Li and Jiefu Zhang and Haixiang Sun and Zhou Li and Vaneet Aggarwal},
howpublished = {arXiv},
year = {2026},
month = aug,
doi = {10.48550/arxiv.2608.30916},
eprint = {2608.30916},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2608.30916},
}
RIS
TY - UNPB
TI - Selection-Aware Stress Testing for Interactive Agents
AU - Yang Xu
AU - Chenang Li
AU - Jiefu Zhang
AU - Haixiang Sun
AU - Zhou Li
AU - Vaneet Aggarwal
PY - 2026
DA - 2026/08/31
PB - arXiv
DO - 10.48550/arxiv.2608.30916
UR - https://arxiv.org/abs/2608.30916
AB - Agent evaluations often use one benchmark to choose a workflow and then search for task types where its advantage weakens, so both conclusions are selected from the same data. We introduce Selection-Aware Semantic Stress Testing (\SASST{}), which learns a task reweighting from pre-execution features on discovery tasks and evaluates the same paired comparison on separate confirmation tasks. The protocol checks support and stability, uses joint bounds for all planned claims, and can return no claim. We prove conditional asymptotic validity under stated cluster assumptions. A forty-cluster audit finds Gaussian undercoverage and conservative Bonferroni t bounds. In one 480-episode τ-bench study, a $3.75$ point discovery gain vanished on confirmation. A second-model study likewise confirmed neither a workflow benefit nor a stable stress rule.
ER -