Evaluation of replication success: A statistical perspective and extended analysis of the SCORE data
- 1Department of Methodology and Statistics, Tilburg University, Tilburg, Netherlands
- 2Universidad de Murcia, Murcia, Spain
Abstract
Replications are more and more conducted in multiple research fields. A recent large-scale replication project, Systematizing Confidence in Open Research and Evidence (SCORE) project, replicated 274 effects from research on business, economics, education, political science, psychology, and sociology. An open question remains how replication success can best be assessed. Thirteen methods for assessing replication success were applied in the SCORE project. The first goal of this paper is studying the statistical properties of replication success methods by a numerical study tailored to the characteristics of the SCORE data. The results of the numerical study show that replication success methods are unable to reliably draw a conclusion about replication success for typical sample sizes of the original study and replication. Hence, there is often insufficient information in one original study and one replication for drawing definite conclusions. The second goal of this paper is taking a novel perspective on the SCORE data by estimating the effect size of an original study and replication. Estimators based on meta-analysis methods that correct for potential bias in the original study were close to unbiased when synthesizing the original study and replication. The reanalysis of the SCORE data revealed that meta-analytic estimates were substantially closer to the effect size of the replication compared to the original study. The null hypothesis of no effect was rejected by the meta-analysis methods in 59.2% to 81.3% of the pairs of an original study and replication. We recommend researchers to make sure that methods for analyzing replicability have favorable statistical properties for the data at hand before applying these methods.
Related
- A Statistical Perspective on Knowledge Distillation: Foundations, Classical Methods, and Large Language Model Extensions
- Comparison of statistical methods for high-dimensional compositional data from flow cytometry: A critical perspective on log-ratio transformation and differential abundance testing
- Statistical Foundations of Reinforcement Learning: A Non-Asymptotic Perspective
- Statistical Methods for Multiple Language Model Comparison on a Shared Evaluation
- Anytime Solver Evaluation with a Normalized Signed Primal Integral and Explicit Reference Policies - Extended Version
BibTeX
@article{aert2026evaluation,
title = {{Evaluation of replication success: A statistical perspective and extended analysis of the SCORE data}},
author = {Robbie C. M. van Aert and Marcel A.L.M. van Assen and Rubén López-Nicolás},
journal = {PLOS One},
year = {2026},
month = sep,
doi = {10.1371/journal.pone.0358126},
url = {https://doi.org/10.1371/journal.pone.0358126},
note = {CC BY 4.0},
}
RIS
TY - JOUR
TI - Evaluation of replication success: A statistical perspective and extended analysis of the SCORE data
AU - Robbie C. M. van Aert
AU - Marcel A.L.M. van Assen
AU - Rubén López-Nicolás
PY - 2026
DA - 2026/09/30
JO - PLOS One
DO - 10.1371/journal.pone.0358126
UR - https://doi.org/10.1371/journal.pone.0358126
AB - Replications are more and more conducted in multiple research fields. A recent large-scale replication project, Systematizing Confidence in Open Research and Evidence (SCORE) project, replicated 274 effects from research on business, economics, education, political science, psychology, and sociology. An open question remains how replication success can best be assessed. Thirteen methods for assessing replication success were applied in the SCORE project. The first goal of this paper is studying the statistical properties of replication success methods by a numerical study tailored to the characteristics of the SCORE data. The results of the numerical study show that replication success methods are unable to reliably draw a conclusion about replication success for typical sample sizes of the original study and replication. Hence, there is often insufficient information in one original study and one replication for drawing definite conclusions. The second goal of this paper is taking a novel perspective on the SCORE data by estimating the effect size of an original study and replication. Estimators based on meta-analysis methods that correct for potential bias in the original study were close to unbiased when synthesizing the original study and replication. The reanalysis of the SCORE data revealed that meta-analytic estimates were substantially closer to the effect size of the replication compared to the original study. The null hypothesis of no effect was rejected by the meta-analysis methods in 59.2% to 81.3% of the pairs of an original study and replication. We recommend researchers to make sure that methods for analyzing replicability have favorable statistical properties for the data at hand before applying these methods.
SN - 1932-6203
ER -