Anytime-Valid Inference in Conjoint Experiments
Abstract
Choosing an appropriate sample size is a fundamental challenge in experimental design. This challenge is particularly salient in conjoint experiments, where researchers simultaneously target many estimands whose inferential goals may be reached at different sample sizes. Conventional sample-size calculations address this challenge in an ex ante manner. In large online studies, including many conjoint experiments, data often arrive continuously, allowing researchers to dynamically stop data collection. In such settings, the relevant practical question is not "what sample size should I choose in advance?" but rather, as data accumulate, "have I learned enough yet?" A naive response is to repeatedly inspect conventional confidence intervals and p-values, but doing so invalidates their inferential guarantees. We address this challenge by developing a general anytime-valid framework for conjoint experiments that allows continuous monitoring and data-dependent stopping. This framework equips many common conjoint estimands with asymptotic e-processes, confidence sequences, and anytime-valid p-values, and provides a general recipe for any estimand that can be expressed as a linear contrast of population OLS coefficients. Across common conjoint settings, we demonstrate that anytime-valid inference can yield substantial sample savings relative to fixed-sample designs. The probability of early stopping frequently exceeds 50%, with mean sample savings often surpassing 25%.
Related
- A Tutorial on Safe Anytime-Valid Inference: Practical Maximally Flexible Sampling Designs for Experiments Based on e-Values
- Anytime-Valid Inference in Linear Models with Applications to Regression-Adjusted Causal Inference
- CAQ: Clustered Asymptotically Anytime-Valid Quantile Treatment Effects for Online Experiments
- Asymptotic Anytime-Valid Quantile Inference under Local Differential Privacy
- Target-Aware Sequential Inference: Pooled versus Stratified Anytime-Valid Designs
BibTeX
@misc{molitor2026anytimevalid,
title = {{Anytime-Valid Inference in Conjoint Experiments}},
author = {Daniel Molitor and Jennah Gosciak and Michael Lindon},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.28979},
eprint = {2610.28979},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.28979},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Anytime-Valid Inference in Conjoint Experiments
AU - Daniel Molitor
AU - Jennah Gosciak
AU - Michael Lindon
PY - 2026
DA - 2026/10/20
PB - arXiv
DO - 10.48550/arxiv.2610.28979
UR - https://arxiv.org/abs/2610.28979
AB - Choosing an appropriate sample size is a fundamental challenge in experimental design. This challenge is particularly salient in conjoint experiments, where researchers simultaneously target many estimands whose inferential goals may be reached at different sample sizes. Conventional sample-size calculations address this challenge in an ex ante manner. In large online studies, including many conjoint experiments, data often arrive continuously, allowing researchers to dynamically stop data collection. In such settings, the relevant practical question is not "what sample size should I choose in advance?" but rather, as data accumulate, "have I learned enough yet?" A naive response is to repeatedly inspect conventional confidence intervals and p-values, but doing so invalidates their inferential guarantees. We address this challenge by developing a general anytime-valid framework for conjoint experiments that allows continuous monitoring and data-dependent stopping. This framework equips many common conjoint estimands with asymptotic e-processes, confidence sequences, and anytime-valid p-values, and provides a general recipe for any estimand that can be expressed as a linear contrast of population OLS coefficients. Across common conjoint settings, we demonstrate that anytime-valid inference can yield substantial sample savings relative to fixed-sample designs. The probability of early stopping frequently exceeds 50%, with mean sample savings often surpassing 25%.
ER -