A Tutorial on Safe Anytime-Valid Inference: Practical Maximally Flexible Sampling Designs for Experiments Based on e-Values
Abstract
We demonstrate how e-values simplify both experimental design and the inference process. With e-values researchers can perform anytime-valid tests and construct confidence intervals that maintain type I error control regardless of the sample size. This enables real-time monitoring of evidence as data are collected, permitting early termination of experiments without intolerably inflating the risk of false discoveries. Early stopping not only con- serves resources, but also mitigates risk for participants in clinical settings. Anytime-valid tests allow for optional continuation, that is, the extension of an experiment, for instance if more funds become available, or even if the evidence looks promising and the funding agency, a reviewer, or an editor urges the experimenter to collect more data. Analogously, a researcher can be assured that a 95% anytime-valid confidence interval will, with at least 95% probability, cover the true effect size regardless of how, or even if, data collection is stopped. We use the free and open-source software package safestats implemented in R to illustrate the practical benefits of this novel inference framework.
Related
- Bayesian Hierarchical Models for Longitudinal Meta-Analysis in Sports Science: A Practical Tutorial Using brms
- Using Customised LLM-Based Chatbots in Psychological Research: A Practical Tutorial
- Flexible nonparametric inference for causal effects under the front-door model
- Steepest Guidance: A Practical and Principled Approach to Inference-Time Alignment of Flow and Diffusion-based Models
- Flexible Doubly Robust Proximal Causal Inference for Censored Outcomes
BibTeX
@misc{ly2026tutorial,
title = {{A Tutorial on Safe Anytime-Valid Inference: Practical Maximally Flexible Sampling Designs for Experiments Based on e-Values}},
author = {Alexander Ly and Udo Boehm and Peter Grünwald and Aaditya Ramdas and Don van Ravenzwaaij},
howpublished = {PsyArXiv},
year = {2026},
month = oct,
doi = {10.31234/osf.io/h5vae_v4},
url = {https://osf.io/preprints/psyarxiv/h5vae_v4/},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - A Tutorial on Safe Anytime-Valid Inference: Practical Maximally Flexible Sampling Designs for Experiments Based on e-Values
AU - Alexander Ly
AU - Udo Boehm
AU - Peter Grünwald
AU - Aaditya Ramdas
AU - Don van Ravenzwaaij
PY - 2026
DA - 2026/10/22
PB - PsyArXiv
DO - 10.31234/osf.io/h5vae_v4
UR - https://osf.io/preprints/psyarxiv/h5vae_v4/
AB - We demonstrate how e-values simplify both experimental design and the inference process. With e-values researchers can perform anytime-valid tests and construct confidence intervals that maintain type I error control regardless of the sample size. This enables real-time monitoring of evidence as data are collected, permitting early termination of experiments without intolerably inflating the risk of false discoveries. Early stopping not only con- serves resources, but also mitigates risk for participants in clinical settings. Anytime-valid tests allow for optional continuation, that is, the extension of an experiment, for instance if more funds become available, or even if the evidence looks promising and the funding agency, a reviewer, or an editor urges the experimenter to collect more data. Analogously, a researcher can be assured that a 95% anytime-valid confidence interval will, with at least 95% probability, cover the true effect size regardless of how, or even if, data collection is stopped. We use the free and open-source software package safestats implemented in R to illustrate the practical benefits of this novel inference framework.
ER -