AI-assisted design and analysis of experiments with unstructured treatments
Abstract
Randomized experiments involving complex, unstructured treatments such as text, images, or video, are increasingly common in social science research. The goal of many of these studies is to estimate the effect of a structured, interpretable attribute that generalizes beyond the treatments used in the experiment (e.g., the presence of a linguistic or visual feature). There are two key challenges when conducting such studies: observed effects of the attribute of interest (i) may be confounded by other correlated attributes, leading to poor internal validity; and (ii) may not generalize beyond the treatments used in the experiment, leading to poor external validity. We propose a framework for designing experiments with unstructured treatments that leverages generative AI or large databases to sample treatments in a way that explicitly navigates the tradeoff between internal and external validity. We show how to conduct conservative asymptotic inference with a difference-in-means estimator, and develop a calibrated AI-assisted estimator that incorporates zero-shot AI predictions to improve precision without introducing bias. Throughout, we work in a finite population framework, where treatment assignment is the only source of randomness, and treat AI generations and predictions as black-boxes with no guarantees on their performance. We illustrate and evaluate our approach by designing two hypothetical experiments, one with text and another with images.
Related
- Network Experiments with Edge Treatments and Node Outcomes
- Identifying Causal Effects in Experiments with Latent Treatments: An Instrumental Variable Approach
- Fisher's Ideas and the Design of Field Experiments in Agronomy and Plant Breeding
- Design-Based Confidence Sequences: A General Approach to Risk Mitigation in Panel Experiments
- Design-based effect sizes and their synthesis in randomized multiple-baseline experiments
BibTeX
@misc{benmichael2026aiassisted,
title = {{AI-assisted design and analysis of experiments with unstructured treatments}},
author = {Eli Ben-Michael and Zach Branson},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.33818},
eprint = {2610.33818},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.33818},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - AI-assisted design and analysis of experiments with unstructured treatments
AU - Eli Ben-Michael
AU - Zach Branson
PY - 2026
DA - 2026/10/24
PB - arXiv
DO - 10.48550/arxiv.2610.33818
UR - https://arxiv.org/abs/2610.33818
AB - Randomized experiments involving complex, unstructured treatments such as text, images, or video, are increasingly common in social science research. The goal of many of these studies is to estimate the effect of a structured, interpretable attribute that generalizes beyond the treatments used in the experiment (e.g., the presence of a linguistic or visual feature). There are two key challenges when conducting such studies: observed effects of the attribute of interest (i) may be confounded by other correlated attributes, leading to poor internal validity; and (ii) may not generalize beyond the treatments used in the experiment, leading to poor external validity. We propose a framework for designing experiments with unstructured treatments that leverages generative AI or large databases to sample treatments in a way that explicitly navigates the tradeoff between internal and external validity. We show how to conduct conservative asymptotic inference with a difference-in-means estimator, and develop a calibrated AI-assisted estimator that incorporates zero-shot AI predictions to improve precision without introducing bias. Throughout, we work in a finite population framework, where treatment assignment is the only source of randomness, and treat AI generations and predictions as black-boxes with no guarantees on their performance. We illustrate and evaluate our approach by designing two hypothetical experiments, one with text and another with images.
ER -