Stand-In Effects and Pooling Amplification: Design-Based Tests of Whether Biological World Models Use Actions
Abstract
Action-conditioned world models are increasingly trained on data from randomized experiments, and whether their predictions respond to actions is usually checked by replacing or removing the actions and measuring the loss in accuracy. We show that such diagnostics report action effects where there are none: under autocorrelated designs, future actions stand in for history the model does not see, and a better model learns the stand-in better. When the design law is known, a conditional randomization test written as an e-value is valid in finite samples whatever the model, provided the resampler conditions on the full state the design used; resampling from a simpler law reproduces the false effect. We identify a second failure, specific to world models: when many sites share one action sequence (cells in a microscope field, pixels in a frame), the natural pooled score turns small model errors along the action contrast into overwhelming evidence against a true effect. In a pixel world, its median log e-value fell from −17.3 to −6832 as frames grew from 64 to 4096 pixels, while the oracle's rose. Tempering the score by a design effect estimated on training data changes the bet, not the validity. Our own preregistered analysis of optogenetic ERK experiments failed in exactly this way; a tempered statistic, selected on synthetic data and registered before the data were obtained, then rejected the null on a never-opened field (E≈ 1.1×1010). Every analysis is preregistered, fixed in a dated protocol before it was run, or labelled exploratory, and every reported result is generated from hashed result files.
Related
- Efficient biological design via genomic in-context learning
- Conditional Proximity Labeling in Complex Biological Systems: Design Principles, Emerging Platforms, and Applications
- Design, synthesis and biological evaluation of pleiotropic prodrugs activated by cholinesterases and releasing a 5-HT4R agonist, with in vivo antiamnesic activity
- Extracellular matrix ageing as a biological variable in 3D cancer models: A framework for age-aware model design
- Whether, or under what conditions? A configurational approach to AI-supported antimicrobial stewardship in sub-Saharan Africa
BibTeX
@misc{xu2026standin,
title = {{Stand-In Effects and Pooling Amplification: Design-Based Tests of Whether Biological World Models Use Actions}},
author = {Bowen Xu and Boyu Chen},
howpublished = {arXiv},
year = {2026},
month = nov,
doi = {10.48550/arxiv.2611.01666},
eprint = {2611.01666},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2611.01666},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Stand-In Effects and Pooling Amplification: Design-Based Tests of Whether Biological World Models Use Actions
AU - Bowen Xu
AU - Boyu Chen
PY - 2026
DA - 2026/11/01
PB - arXiv
DO - 10.48550/arxiv.2611.01666
UR - https://arxiv.org/abs/2611.01666
AB - Action-conditioned world models are increasingly trained on data from randomized experiments, and whether their predictions respond to actions is usually checked by replacing or removing the actions and measuring the loss in accuracy. We show that such diagnostics report action effects where there are none: under autocorrelated designs, future actions stand in for history the model does not see, and a better model learns the stand-in better. When the design law is known, a conditional randomization test written as an e-value is valid in finite samples whatever the model, provided the resampler conditions on the full state the design used; resampling from a simpler law reproduces the false effect. We identify a second failure, specific to world models: when many sites share one action sequence (cells in a microscope field, pixels in a frame), the natural pooled score turns small model errors along the action contrast into overwhelming evidence against a true effect. In a pixel world, its median log e-value fell from −17.3 to −6832 as frames grew from 64 to 4096 pixels, while the oracle's rose. Tempering the score by a design effect estimated on training data changes the bet, not the validity. Our own preregistered analysis of optogenetic ERK experiments failed in exactly this way; a tempered statistic, selected on synthetic data and registered before the data were obtained, then rejected the null on a never-opened field (E≈ 1.1×10^10). Every analysis is preregistered, fixed in a dated protocol before it was run, or labelled exploratory, and every reported result is generated from hashed result files.
ER -