Using an open science checklist in grant proposal reviews to predict reproducibility of funded publications
Abstract
Objectives: To determine whether an open science checklist can be useful for predicting the reproducibility of publications resulting from these grant proposals when used by grant referees assessing them. Study design and setting: This is a comparative accuracy study design using funded grant proposals obtained from online sources (i.e., Open Grants, RIO Journal, NIH, and Grantome). Two independent groups of mock referees assessed open science practices in the proposals and predicted whether the resulting publications would be reproducible, with one group using an open science checklist as an intervention, and the other without. Then we attempted to reproduce the primary findings of a resulting publication from the grant proposal. Sensitivity, specificity, predictive values, and overall accuracy were calculated from 2 × 2 tables, comparing predicted vs. actual reproducibility. The primary outcome is the level of reproducibility, measured by the predictive value, the proportion of (non)reproducible study findings that are accurately predicted. This study was conducted between April and September 2025. Results: Of seven out of 101 publications (6.9%, 95% CI: 2.8%-13.8%), the primary results could be reproduced. When using the checklist, only 16.8% of the proposals were expected to be reproducible, whereas without the checklist 75.2% were expected to be reproducible. When using the checklist, 17 proposals were expected to be reproducible, whereas only 2 out of these 17 could actually be reproduced (positive predictive value (PPV) 11.8% (95% CI: 3.3%-34.3%). Without using the checklist, 76 proposals were thought to be reproducible, whereas only six out of 76 could actually be reproduced (PPV 7.9% [95% CI: 3.7-16.2%]). Sensitivity analysis by research field was not conducted because of small sample sizes in most categories. Conclusion: The open science checklist has a low positive predictive value, as expected given the low reproducibility prevalence in our sample. Although the differences between the group using the checklist and the group that did not use the checklist may also have been caused by their level of knowledge of reproducibility and open science, neither group could predict which proposals would or would not be reproducible.
Related
- Alien Vascular Plants of Kazakhstan: The First National Checklist and Spatial Distribution Patterns
- Alien ornamental plants spontaneously spreading in Eretz-Israel of mid-20th century: review of forgotten publications
- Occurrence of Yunnan Black-veined White Aporia goutellei (Oberthür, 1886) (Lepidoptera: Pieridae: Pierinae: Pierini) in Anjaw District of Arunachal Pradesh: a new addition to the Indian butterfly checklist
- The treehoppers (Hemiptera, Membracidae) of Singapore: An annotated checklist, new records, and an illustrated identification key
- Checklist of Korean Mymaridae (Hymenoptera, Chalcidoidea) with new records and descriptions of two new species of Zeyanus from South Korea and Japan.
BibTeX
@article{dewi2026using,
title = {{Using an open science checklist in grant proposal reviews to predict reproducibility of funded publications}},
author = {Ayu Putu Madri Dewi and Anne Floor Holtrop and Nicholas J Devito and Melissa L. Rethlefsen and Zulfan Zazuli and Mathieu Repiquet and Ka Hin Tai and Constant Vinatier and Monika Varga and Stefan Wouters and Hengky Kurniawan and Nynke Swart and Gowri Gopalakrishna and Inge Stegeman and Mariska M Leeflang},
journal = {Journal of Clinical Epidemiology},
year = {2026},
month = oct,
doi = {10.1016/j.jclinepi.2026.112399},
url = {https://hal.science/hal-05752362v1},
note = {CC BY 4.0},
}
RIS
TY - JOUR
TI - Using an open science checklist in grant proposal reviews to predict reproducibility of funded publications
AU - Ayu Putu Madri Dewi
AU - Anne Floor Holtrop
AU - Nicholas J Devito
AU - Melissa L. Rethlefsen
AU - Zulfan Zazuli
AU - Mathieu Repiquet
AU - Ka Hin Tai
AU - Constant Vinatier
AU - Monika Varga
AU - Stefan Wouters
AU - Hengky Kurniawan
AU - Nynke Swart
AU - Gowri Gopalakrishna
AU - Inge Stegeman
AU - Mariska M Leeflang
PY - 2026
DA - 2026/10/19
JO - Journal of Clinical Epidemiology
DO - 10.1016/j.jclinepi.2026.112399
UR - https://hal.science/hal-05752362v1
AB - Objectives: To determine whether an open science checklist can be useful for predicting the reproducibility of publications resulting from these grant proposals when used by grant referees assessing them. Study design and setting: This is a comparative accuracy study design using funded grant proposals obtained from online sources (i.e., Open Grants, RIO Journal, NIH, and Grantome). Two independent groups of mock referees assessed open science practices in the proposals and predicted whether the resulting publications would be reproducible, with one group using an open science checklist as an intervention, and the other without. Then we attempted to reproduce the primary findings of a resulting publication from the grant proposal. Sensitivity, specificity, predictive values, and overall accuracy were calculated from 2 × 2 tables, comparing predicted vs. actual reproducibility. The primary outcome is the level of reproducibility, measured by the predictive value, the proportion of (non)reproducible study findings that are accurately predicted. This study was conducted between April and September 2025. Results: Of seven out of 101 publications (6.9%, 95% CI: 2.8%-13.8%), the primary results could be reproduced. When using the checklist, only 16.8% of the proposals were expected to be reproducible, whereas without the checklist 75.2% were expected to be reproducible. When using the checklist, 17 proposals were expected to be reproducible, whereas only 2 out of these 17 could actually be reproduced (positive predictive value (PPV) 11.8% (95% CI: 3.3%-34.3%). Without using the checklist, 76 proposals were thought to be reproducible, whereas only six out of 76 could actually be reproduced (PPV 7.9% [95% CI: 3.7-16.2%]). Sensitivity analysis by research field was not conducted because of small sample sizes in most categories. Conclusion: The open science checklist has a low positive predictive value, as expected given the low reproducibility prevalence in our sample. Although the differences between the group using the checklist and the group that did not use the checklist may also have been caused by their level of knowledge of reproducibility and open science, neither group could predict which proposals would or would not be reproducible.
SN - 0895-4356
SN - 1878-5921
ER -