Defining and Differentiating Repeatability in Monte Carlo Simulation Studies
Abstract
Monte Carlo simulation studies are a primary methodology used to evaluate statistical methods which then are used in psychology and other social-behavioral research. While the replication crisis in social-behavioral science led to deep reflection on the repeatability of research studies and findings, the same is just beginning for Monte Carlo simulation studies. Recent evidence suggests there are reasons for concern about the repeatability of Monte Carlo simulations, which could have major implications for social-behavioral sciences more broadly, given the impact these studies have on analytical approaches used across fields. In this paper, we adopt a specific approach to describing and differentiating repeatability terms (replication, reproduction, and robustness; Nosek et al., 2025) and explore how these concepts should be applied and revised for the context of Monte Carlo simulations. We suggest key deviations from the Nosek et al. (2025) framework and a subdivision of repeatability types stemming from the potential for more systematic exploration of the role of sampling variability in simulation research through the sharing of open code. This approach demonstrates the importance and distinct value of each type of repeatability, as well as identifies the role of open data and code, and other verification studies, like multiverse analyses and registered reports. Monte Carlo simulation studies are important for informing statistical practice, and ensuring the repeatability of these studies is tantamount to the appropriate use of statistical methods in psychology and beyond.
Related
- Probabilistic clinical studies and deterministic physiology: building a Monte Carlo bridge with the "Frank" project.
- A Kinetic Theory Approach To Ordered Fluids: Direct Simulation Monte Carlo Methods
- Development of an efficient neutrosophic estimator for finite population mean using Monte Carlo simulation and multi-domain data
- Probabilistic health risk assessment of respirable dust exposure among stone carvers in Thailand using average daily dose and Monte Carlo simulation
- Bridging Markov Chain Monte Carlo Techniques and Tierney–Kadane Approximations for Progressively Censored Garhy Reliability Models: Simulation Insights and a Medical Application
BibTeX
@misc{montoya2026defining,
title = {{Defining and Differentiating Repeatability in Monte Carlo Simulation Studies}},
author = {Amanda Montoya and Samantha F. Anderson},
howpublished = {PsyArXiv},
year = {2026},
month = sep,
doi = {10.31234/osf.io/psfkt_v2},
url = {https://osf.io/preprints/psyarxiv/psfkt_v2/},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Defining and Differentiating Repeatability in Monte Carlo Simulation Studies
AU - Amanda Montoya
AU - Samantha F. Anderson
PY - 2026
DA - 2026/09/05
PB - PsyArXiv
DO - 10.31234/osf.io/psfkt_v2
UR - https://osf.io/preprints/psyarxiv/psfkt_v2/
AB - Monte Carlo simulation studies are a primary methodology used to evaluate statistical methods which then are used in psychology and other social-behavioral research. While the replication crisis in social-behavioral science led to deep reflection on the repeatability of research studies and findings, the same is just beginning for Monte Carlo simulation studies. Recent evidence suggests there are reasons for concern about the repeatability of Monte Carlo simulations, which could have major implications for social-behavioral sciences more broadly, given the impact these studies have on analytical approaches used across fields. In this paper, we adopt a specific approach to describing and differentiating repeatability terms (replication, reproduction, and robustness; Nosek et al., 2025) and explore how these concepts should be applied and revised for the context of Monte Carlo simulations. We suggest key deviations from the Nosek et al. (2025) framework and a subdivision of repeatability types stemming from the potential for more systematic exploration of the role of sampling variability in simulation research through the sharing of open code. This approach demonstrates the importance and distinct value of each type of repeatability, as well as identifies the role of open data and code, and other verification studies, like multiverse analyses and registered reports. Monte Carlo simulation studies are important for informing statistical practice, and ensuring the repeatability of these studies is tantamount to the appropriate use of statistical methods in psychology and beyond.
ER -