Can AI Agents Simulate A/B Test Outcomes? A Validation Framework for Agentic Experimentation
Abstract
A/B testing remains the standard for rolling out new features in the technology industry. Each experiment, however, consumes real traffic, engineering effort, and weeks of wall-clock time. Can AI agents---conditioned on behavioral profiles and contextual descriptions of the intervention---simulate outcomes accurately enough to vet candidate treatments before committing live traffic? We formalize this question as a \emph{Simulated Randomized Controlled Trial} (S-RCT) and derive a two-layer error decomposition that separates agent approximation error from subsampling error, enabling targeted improvements to each. The framework is agent-agnostic: any behavioral model---from a fine-tuned specialist to a general-purpose foundation model---can serve as the simulation engine. Validated on 67 historical marketing A/B tests, a baseline S-RCT using an off-the-shelf foundation model captures directional signal (sign overlap 0.70) but systematically overshoots effect magnitudes. A two-phase pre-period calibration protocol reduces the squared prediction error (after removing irreducible measurement noise) by ∼77×; a within-subject design---where each agent is exposed to both arms---reduces standard errors by ∼2.4×. We discuss limitations of the current approach and identify applications where experimenters stand to benefit from agentic signals.
Related
- From Test Performance to Risk-Based Effect Sizes: A Unified Wald-Type Framework to Design Clinical Validation Studies for Binary and Survival Outcomes
- Governing Agentic AI in Africa: A Systematic Literature Review of Ethical Challenges, Framework Gaps, and Ubuntu-Informed Governance Solutions
- CausalSmith: A Formally Grounded, Self-Improving Agentic Framework for Automated Research in Causal Inference
- On the Multi-Index, Multi-Rate, and Multi-Phase Dynamics of Decoder-Only Language Models: A Unified Hybrid Framework for Generative and Agentic Systems
- Solver Agent: an Agentic AI Framework for Theoretical Physics Computations Applied to F-theory Uplifts of O3-planes and S-folds
BibTeX
@misc{hut2026agents,
title = {{Can AI Agents Simulate A/B Test Outcomes? A Validation Framework for Agentic Experimentation}},
author = {Stefan Hut and Lorenzo Masoero},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2608.02345},
eprint = {2608.02345},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2608.02345},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Can AI Agents Simulate A/B Test Outcomes? A Validation Framework for Agentic Experimentation
AU - Stefan Hut
AU - Lorenzo Masoero
PY - 2026
DA - 2026/09/11
PB - arXiv
DO - 10.48550/arxiv.2608.02345
UR - https://arxiv.org/abs/2608.02345
AB - A/B testing remains the standard for rolling out new features in the technology industry. Each experiment, however, consumes real traffic, engineering effort, and weeks of wall-clock time. Can AI agents---conditioned on behavioral profiles and contextual descriptions of the intervention---simulate outcomes accurately enough to vet candidate treatments before committing live traffic? We formalize this question as a \emph{Simulated Randomized Controlled Trial} (S-RCT) and derive a two-layer error decomposition that separates agent approximation error from subsampling error, enabling targeted improvements to each. The framework is agent-agnostic: any behavioral model---from a fine-tuned specialist to a general-purpose foundation model---can serve as the simulation engine. Validated on 67 historical marketing A/B tests, a baseline S-RCT using an off-the-shelf foundation model captures directional signal (sign overlap 0.70) but systematically overshoots effect magnitudes. A two-phase pre-period calibration protocol reduces the squared prediction error (after removing irreducible measurement noise) by ∼77×; a within-subject design---where each agent is exposed to both arms---reduces standard errors by ∼2.4×. We discuss limitations of the current approach and identify applications where experimenters stand to benefit from agentic signals.
ER -