Adaptive Experimentation Can Hide Model Error:Detection Limits and Falsification Budget
Abstract
An experiment can be informative about the parameters of a working model and uninformative about whether that model is wrong. Adaptive design can reinforce this mismatch by repeatedly selecting settings where an incorrect model fits. We bound the power of any valid adequacy test in terms of the experimenter's own sampling distribution, then distinguish information absent from the selected data from information lost by the detector. A computable diagnostic measures unresolved error relative to a declared expansion of the working model. It motivates a training-free rule that reserves part of the design budget for falsification. The empirical record is mixed: the independently specified primary study fails detection eligibility, and a later numerator diagnostic fails its improvement criteria. After the frozen rule fails on fresh continuous-null panels, a disclosed secondary horizon-calibrated revision detects 35.8% of misspecified campaigns versus 31.7% for a class-separation control at T = 240, but has no resolved advantage at T = 480. In a corrected 289-unit measured replay it improves on random design and trails class-discrimination EIG. A conditional futility bound explains why remaining time matters without proving the gate constant or its power. These results support separating parameter learning, model checking and detector performance; they do not establish a universally superior design policy or prospective scientific discovery.
Related
- Principal-agent hypothesis testing under adaptive experimentation
- Sharp Limits for Honest Uncertainty in Hard-Budget Repeated Evaluation
- Adaptive Multi Mode: Budget-Aware Routing of Optimizers By Online Landscape Probes
- Regret-based Characterization, Information Limits, and the Sample Complexity of Auditing Adaptive Policies
- Adaptive control of pool-type research nuclear reactors: theory limits, pragmatic actuation constraints, and realized performance and adaptation
BibTeX
@misc{choudhary2026adaptive,
title = {{Adaptive Experimentation Can Hide Model Error:Detection Limits and Falsification Budget}},
author = {Ramnarayan Choudhary},
howpublished = {arXiv},
year = {2026},
month = nov,
doi = {10.48550/arxiv.2611.07529},
eprint = {2611.07529},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2611.07529},
note = {CC BY-NC-ND 4.0},
}
RIS
TY - UNPB
TI - Adaptive Experimentation Can Hide Model Error:Detection Limits and Falsification Budget
AU - Ramnarayan Choudhary
PY - 2026
DA - 2026/11/05
PB - arXiv
DO - 10.48550/arxiv.2611.07529
UR - https://arxiv.org/abs/2611.07529
AB - An experiment can be informative about the parameters of a working model and uninformative about whether that model is wrong. Adaptive design can reinforce this mismatch by repeatedly selecting settings where an incorrect model fits. We bound the power of any valid adequacy test in terms of the experimenter's own sampling distribution, then distinguish information absent from the selected data from information lost by the detector. A computable diagnostic measures unresolved error relative to a declared expansion of the working model. It motivates a training-free rule that reserves part of the design budget for falsification. The empirical record is mixed: the independently specified primary study fails detection eligibility, and a later numerator diagnostic fails its improvement criteria. After the frozen rule fails on fresh continuous-null panels, a disclosed secondary horizon-calibrated revision detects 35.8% of misspecified campaigns versus 31.7% for a class-separation control at T = 240, but has no resolved advantage at T = 480. In a corrected 289-unit measured replay it improves on random design and trails class-discrimination EIG. A conditional futility bound explains why remaining time matters without proving the gate constant or its power. These results support separating parameter learning, model checking and detector performance; they do not establish a universally superior design policy or prospective scientific discovery.
ER -