Skip to content
DraconiticResearch
Search Sources API
203,234 studies31 sourcesUpdated 1 h ago
PreprintarXiv licencearXiv

Always-On Experimentation

  • Ricardo J. Sandoval
  • David Arbour
  • Avi Feller
  • Michael I. Jordan
Open at sourcePDFDOICite

Abstract

Generative AI has dramatically accelerated the rate at which new treatments---from novel pharmaceuticals to online marketing campaigns---can be conceived and deployed. As a result, modern experimentation platforms often run continuously, with treatments added as they are ready and removed when they underperform. We formalize this "Always-On" experimental setting, in which treatments can be dynamically generated, added to, and removed from a running experiment, and study the statistical problem of deciding whether to accept or reject each treatment while controlling for the false discovery rate. We develop sequential tests that achieve time-uniform Type-I error control under arbitrary stopping times and "predictable" treatment schedules. Our approach builds on the testing-by-betting framework: we construct test supermartingales for testing the average treatment effect of each treatment, and show that the construction of these test supermartingales is growth-rate optimal in an almost-sure sense.

Related

  1. Un module universitaire de type « capstone course » en arithmétique – une pré-expérimentation
    Monique Chappet-Paries, Françoise Pilorge, Aline Robert +1
    HALPreprintCC BY-NC-ND 4.0Full textGeneral mathematicsHAL
  2. Atomic Leontievian Cournotian traders are always Walrasian
    Francesca Busetto, Giulio Codognato, Ludovic Julien
    Games and Economic BehaviorPeer-reviewedLicence not statedFull text4 citedGeneral mathematicsHAL
  3. The product of two Scott sober complete lattices is not always Scott sober
    Zhengmao He
    arXivPreprintCC BY 4.0Full textGeometry & topologyarXiv
  4. Principal-agent hypothesis testing under adaptive experimentation
    Ben Chugg, Aritra Ghosh, Aaditya Ramdas
    arXivPreprintCC BY 4.0Full textStatisticsarXiv
  5. The entropy production is not always monotone for hard spheres or for Maxwell molecules
    Luis Silvestre
    arXivPreprintarXiv licenceFull textAnalysis & PDEsMathematical physicsarXiv

Cite

BibTeXRIS
BibTeX
@misc{sandoval2026alwayson,
  title = {{Always-On Experimentation}},
  author = {Ricardo J. Sandoval and David Arbour and Avi Feller and Michael I. Jordan},
  howpublished = {arXiv},
  year = {2026},
  month = sep,
  doi = {10.48550/arxiv.2609.38695},
  eprint = {2609.38695},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2609.38695},
}
RIS
TY  - UNPB
TI  - Always-On Experimentation
AU  - Ricardo J. Sandoval
AU  - David Arbour
AU  - Avi Feller
AU  - Michael I. Jordan
PY  - 2026
DA  - 2026/09/30
PB  - arXiv
DO  - 10.48550/arxiv.2609.38695
UR  - https://arxiv.org/abs/2609.38695
AB  - Generative AI has dramatically accelerated the rate at which new treatments---from novel pharmaceuticals to online marketing campaigns---can be conceived and deployed. As a result, modern experimentation platforms often run continuously, with treatments added as they are ready and removed when they underperform. We formalize this "Always-On" experimental setting, in which treatments can be dynamically generated, added to, and removed from a running experiment, and study the statistical problem of deciding whether to accept or reject each treatment while controlling for the false discovery rate. We develop sequential tests that achieve time-uniform Type-I error control under arbitrary stopping times and "predictable" treatment schedules. Our approach builds on the testing-by-betting framework: we construct test supermartingales for testing the average treatment effect of each treatment, and show that the construction of these test supermartingales is growth-rate optimal in an almost-sure sense.
ER  -