Designing Experiments for External Decision Makers
Abstract
Standard Bayesian experimental design (BED) methods assume the experimenter alone will act on the data, choosing the best action given their own beliefs and goals. In practice, data are often acted on by external parties whose procedures the experimenter cannot change and may not know, particularly when the data are released publicly. To extend BED to such settings, we introduce usage-aware experimental design, a framework in which an adaptive design policy is trained to minimise the usage risk: the experimenter's expected loss when one or more external rules act on the collected data. These rules may be non-Bayesian or black-box, and when several may apply, we either average over them or take the worst case. We show that, unlike in standard BED, more informative data can be harmful under such rules, so the usage risk cannot in general be expressed as an expected uncertainty reduction; our framework thus strictly generalises standard BED. We then use it to tackle a range of problems that are difficult to address with existing approaches.
Related
- Are Good Generators Good Decision-Makers? Policy Learning for General Interventions via Retargeted Counterfactual Generation
- Designing multi‐site studies for external validity: Site selection via synthetic purposive sampling
- Covariate Adjustment in Randomized Experiments: A Unified Framework for Decision and Practice
- Optimal Multi-way Decision Trees for Stratified Sampling in Online Controlled Experiments
- Designing Spatial Treatments
BibTeX
@misc{iqbal2026designing,
title = {{Designing Experiments for External Decision Makers}},
author = {Sahel Iqbal and Tom Rainforth},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.22032},
eprint = {2610.22032},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.22032},
}
RIS
TY - UNPB
TI - Designing Experiments for External Decision Makers
AU - Sahel Iqbal
AU - Tom Rainforth
PY - 2026
DA - 2026/10/15
PB - arXiv
DO - 10.48550/arxiv.2610.22032
UR - https://arxiv.org/abs/2610.22032
AB - Standard Bayesian experimental design (BED) methods assume the experimenter alone will act on the data, choosing the best action given their own beliefs and goals. In practice, data are often acted on by external parties whose procedures the experimenter cannot change and may not know, particularly when the data are released publicly. To extend BED to such settings, we introduce usage-aware experimental design, a framework in which an adaptive design policy is trained to minimise the usage risk: the experimenter's expected loss when one or more external rules act on the collected data. These rules may be non-Bayesian or black-box, and when several may apply, we either average over them or take the worst case. We show that, unlike in standard BED, more informative data can be harmful under such rules, so the usage risk cannot in general be expressed as an expected uncertainty reduction; our framework thus strictly generalises standard BED. We then use it to tackle a range of problems that are difficult to address with existing approaches.
ER -