Kernel-based External Validity of Experimental Studies
Abstract
The external validity of a study is the extent to which its findings generalize to unseen experimental conditions, such as datasets. We formalize the results of a study together with its research question, and define external validity as the probability that two independent and same-sized replications of the study produce results whose maximum mean discrepancy (MMD) is below a given tolerance. This approach, centered on the choice of a space of results and of an appropriate reproducing kernel for the MMD, is flexible enough to be deployed on multiple different types of results and research questions. Additionally, we establish an asymptotic power-law relation between the size of a study and the quantiles of the MMD, which we use to estimate the desired size of an externally valid study from a small-scale preliminary study. We deploy our framework to rankings with ties and discuss kernels suited to different research questions. Experiments on existing case studies and on a simulated newly designed study illustrate the applicability of the framework. We release our code as part of the genexpy Python module.
Related
- Designing multi‐site studies for external validity: Site selection via synthetic purposive sampling
- What Pilot Studies Can (and Cannot) Do for Validity in Psychological Research
- Addressing threats to validity in large dataset research
- Long-time validity of the Korteweg-de Vries approximation for the Boussinesq equation
- A counterexample to the pointwise validity of an asymptotic mean value property for p-harmonic functions
BibTeX
@misc{matteucci2026kernelbased,
title = {{Kernel-based External Validity of Experimental Studies}},
author = {Federico Matteucci and Gerhard Neumann},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.13625},
eprint = {2610.13625},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.13625},
}
RIS
TY - UNPB
TI - Kernel-based External Validity of Experimental Studies
AU - Federico Matteucci
AU - Gerhard Neumann
PY - 2026
DA - 2026/10/09
PB - arXiv
DO - 10.48550/arxiv.2610.13625
UR - https://arxiv.org/abs/2610.13625
AB - The external validity of a study is the extent to which its findings generalize to unseen experimental conditions, such as datasets. We formalize the results of a study together with its research question, and define external validity as the probability that two independent and same-sized replications of the study produce results whose maximum mean discrepancy (MMD) is below a given tolerance. This approach, centered on the choice of a space of results and of an appropriate reproducing kernel for the MMD, is flexible enough to be deployed on multiple different types of results and research questions. Additionally, we establish an asymptotic power-law relation between the size of a study and the quantiles of the MMD, which we use to estimate the desired size of an externally valid study from a small-scale preliminary study. We deploy our framework to rankings with ties and discuss kernels suited to different research questions. Experiments on existing case studies and on a simulated newly designed study illustrate the applicability of the framework. We release our code as part of the genexpy Python module.
ER -