Anytime-valid simulation-based hypothesis testing
Abstract
For a given data distribution (Xt)t ∈ ℕ ∼ Q i.i.d., we investigate the hypothesis testing problem: H0: Q = P0 vs. H1: Q = P1, for two different model probability distributions P0 and P1. In contrast to the standard setting, where analytic densities p0 and p1 are given, here, we consider the density-free setting, where we only have access to i.i.d. simulations (Z0t)t ∈ ℕ ∼ P0 and (Z1t)t ∈ ℕ ∼ P1. For this simulation-based hypothesis testing setting, we construct an e-test martingale, resulting in a sequential test with anytime-valid type-I error guarantees, approximate growth optimality, geometrically decaying type-II error bounds, and asymptotic power one. Most ingredients used in our constructions are variants of well known concepts. The value of this paper lies in the compact presentation of an effective, anytime-valid solution for the density-free simulation-based sequential hypothesis testing case.
Related
- Boot-AVT: Asymptotically Anytime-Valid Testing with Degenerate U-Statistics via Bootstrap
- Asymptotic Anytime-Valid Quantile Inference under Local Differential Privacy
- Anytime Valid Confidence Bands for Distribution Functions
- Anytime-valid detection of LLM weight exfiltration
- CAQ: Clustered Asymptotically Anytime-Valid Quantile Treatment Effects for Online Experiments
BibTeX
@misc{forre2026anytimevalid,
title = {{Anytime-valid simulation-based hypothesis testing}},
author = {Patrick Forré and Lydia Brenner},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.08210},
eprint = {2610.08210},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.08210},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Anytime-valid simulation-based hypothesis testing
AU - Patrick Forré
AU - Lydia Brenner
PY - 2026
DA - 2026/10/06
PB - arXiv
DO - 10.48550/arxiv.2610.08210
UR - https://arxiv.org/abs/2610.08210
AB - For a given data distribution (X_t)_t ∈ ℕ ∼ Q i.i.d., we investigate the hypothesis testing problem: H_0: Q = P_0 vs. H_1: Q = P_1, for two different model probability distributions P_0 and P_1. In contrast to the standard setting, where analytic densities p_0 and p_1 are given, here, we consider the density-free setting, where we only have access to i.i.d. simulations (Z^0_t)_t ∈ ℕ ∼ P_0 and (Z^1_t)_t ∈ ℕ ∼ P_1. For this simulation-based hypothesis testing setting, we construct an e-test martingale, resulting in a sequential test with anytime-valid type-I error guarantees, approximate growth optimality, geometrically decaying type-II error bounds, and asymptotic power one. Most ingredients used in our constructions are variants of well known concepts. The value of this paper lies in the compact presentation of an effective, anytime-valid solution for the density-free simulation-based sequential hypothesis testing case.
ER -