Testing structural interpretations beyond goodness-of-fit in a phenomenological epidemic model: A reproducibility-oriented case study
- 1The University of Tokyo
Abstract
Simulation models are widely used to support counterfactual inference in settings where direct experimental validation is difficult or infeasible. However, for phenomenological models calibrated to observational time series, goodness-of-fit does not necessarily establish that the structural assumptions underlying such inferences are uniquely supported by the data. Here, we examine this issue using a published COVID-19 transmission model for Japan that incorporates empirical inputs including human mobility, variant prevalence, and vaccination and has been used to estimate the effects of alternative vaccination scenarios. Using the publicly archived code together with a reconstructed substitute for a restricted-access input file required by the released implementation, we first reproduced the published model behaviour. We then performed two complementary computational stress tests. First, the vaccination-related input was suppressed and the remaining parameters were re-estimated against the original epidemic trajectory. Second, as a complementary test of trajectory-level flexibility, the model was refitted to synthetic target series with qualitatively different temporal structures while retaining the empirical mobility and variant inputs. The model retained substantial fitting capacity in both tests: suppression of the vaccination-related input caused only a limited deterioration in fit to the original trajectory, while qualitatively different synthetic trajectories could also be reproduced through parameter adjustment. Variation in fitted parameters across these conditions was also consistent with compensatory parameter behaviour. These results do not imply that vaccination had no effect or that the original model is invalid. Rather, they provide empirical evidence that, in this case, goodness-of-fit to the observed epidemic trajectory alone does not uniquely support the structural interpretation required for vaccination-related counterfactual inference. The study illustrates the value of reproducible computational stress testing as a complement to, rather than a substitute for, formal identifiability or model-comparison analyses.
Related
- Alternative interpretations of the association between interparental conflict and adolescent online flaming: A reanalysis of Zeng et al. (2026)
- Understanding interpretations of credibility, health literacy and health information seeking in Black and South Asian ethnic communities in the United Kingdom: A qualitative study
- Quantifying Reproducibility of Common Laboratory Tests in a Tertiary Care Hospital for More Realistic Synthetic Data in Machine Learning
- Where did it go? How Information Management Practices Impact Rigor and Reproducibility in Collaborative Clinical and Translational Research
- Beyond Breslow Thickness: Reproducibility and Histopathologic Concordance of Ultra-High-Frequency Ultrasound in Primary Cutaneous Melanoma
BibTeX
@misc{umeno2026testing,
title = {{Testing structural interpretations beyond goodness-of-fit in a phenomenological epidemic model: A reproducibility-oriented case study}},
author = {Y. Umeno},
howpublished = {medRxiv},
year = {2026},
month = oct,
doi = {10.64898/2026.10.13.26365491},
url = {https://www.medrxiv.org/content/10.64898/2026.10.13.26365491v1},
note = {CC BY-NC-ND},
}
RIS
TY - UNPB
TI - Testing structural interpretations beyond goodness-of-fit in a phenomenological epidemic model: A reproducibility-oriented case study
AU - Y. Umeno
PY - 2026
DA - 2026/10/13
PB - medRxiv
DO - 10.64898/2026.10.13.26365491
UR - https://www.medrxiv.org/content/10.64898/2026.10.13.26365491v1
AB - Simulation models are widely used to support counterfactual inference in settings where direct experimental validation is difficult or infeasible. However, for phenomenological models calibrated to observational time series, goodness-of-fit does not necessarily establish that the structural assumptions underlying such inferences are uniquely supported by the data. Here, we examine this issue using a published COVID-19 transmission model for Japan that incorporates empirical inputs including human mobility, variant prevalence, and vaccination and has been used to estimate the effects of alternative vaccination scenarios. Using the publicly archived code together with a reconstructed substitute for a restricted-access input file required by the released implementation, we first reproduced the published model behaviour. We then performed two complementary computational stress tests. First, the vaccination-related input was suppressed and the remaining parameters were re-estimated against the original epidemic trajectory. Second, as a complementary test of trajectory-level flexibility, the model was refitted to synthetic target series with qualitatively different temporal structures while retaining the empirical mobility and variant inputs. The model retained substantial fitting capacity in both tests: suppression of the vaccination-related input caused only a limited deterioration in fit to the original trajectory, while qualitatively different synthetic trajectories could also be reproduced through parameter adjustment. Variation in fitted parameters across these conditions was also consistent with compensatory parameter behaviour. These results do not imply that vaccination had no effect or that the original model is invalid. Rather, they provide empirical evidence that, in this case, goodness-of-fit to the observed epidemic trajectory alone does not uniquely support the structural interpretation required for vaccination-related counterfactual inference. The study illustrates the value of reproducible computational stress testing as a complement to, rather than a substitute for, formal identifiability or model-comparison analyses.
ER -