Core open science practices in major medical journals: development of an automatized tool based on a Large Language Model
Abstract
Background: Open science practices are increasingly promoted to improve the transparency and reproducibility of biomedical research, and their large-scale monitoring has been identified as a priority by initiatives such as the UNESCO Recommendation on Open Science. Existing automated tools cover only narrow subsets of practices. We developed and validated an automated tool, based on locally run open-weights large language models (LLMs), to extract a core set of 13 open science practices from published medical research articles. Methods: We built a validation database of 303 research articles published in 2020–2021 in 10 major general medical journals, stratified across randomized controlled trials (RCTs), meta-analyses, and other designs. Duplicate manual extraction with expert adjudication served as the reference standard. The pipeline combined document conversion (GROBID), BM25-based chunk retrieval, and classification by three Llama models (3-8B, 3-70B, 3.3-70B) run entirely locally, with and without supplementary materials. Diagnostic performance (sensitivity, specificity, predictive values, F1) was computed against the reference standard. Results: Usable outputs were obtained for 301/303 articles. Performance was heterogeneous across criteria rather than across models: practices reported in standardized statements (conflicts of interest, funding, author contributions, data sharing, pre-registration, reporting guidelines) were extracted reliably (F1 ≥ 0.75 for at least one model), whereas rare or inconsistently reported practices (code sharing, statistical analysis plan, protocol sharing, ORCID, open access status, preprints) remained difficult regardless of model size. Study-type classification exceeded 0.90 on all metrics. Larger models improved several criteria at a 6.6–7.5× computational cost, and adding supplementary materials did not improve performance, mainly due to conversion failures. Conclusions: A fully local, open-weights LLM pipeline can reliably monitor a substantial subset of open science practices. Document conversion and layout heterogeneity, rather than language understanding, constitute the main barrier to scaling automated open science monitoring.
Related
- Open Science Policies and Practices Among Medical Journals
- Quadrant classification and development path of gymnastics-related research topics in Chinese high-quality journals: Based on SCI, PKU Core and CSSCI source journals
- Knowledge, attitudes, and prevention practices regarding human papillomavirus among health and social science female students in Northwest Ethiopia: A comparative cross-sectional study
- How science gets by with a little help from the Beatles: Detecting cultural wordplay in scientific literature using large language models
- A practical risk framework for large language model use in life science research
BibTeX
@article{vinatier2026core,
title = {{Core open science practices in major medical journals: development of an automatized tool based on a Large Language Model}},
author = {Constant Vinatier and Guillaume Freyermuth and Margaux Millour and Gauthieu Le Bartz Lyan and Sarah Buet and Perrine Lunel and Nicolas Robillard and Florian Naudet and Mathieu Acher},
journal = {Open Research Europe},
year = {2026},
month = sep,
doi = {10.12688/openreseurope.24615.1},
url = {https://hal.science/hal-05751701v1},
note = {CC BY 4.0},
}
RIS
TY - JOUR
TI - Core open science practices in major medical journals: development of an automatized tool based on a Large Language Model
AU - Constant Vinatier
AU - Guillaume Freyermuth
AU - Margaux Millour
AU - Gauthieu Le Bartz Lyan
AU - Sarah Buet
AU - Perrine Lunel
AU - Nicolas Robillard
AU - Florian Naudet
AU - Mathieu Acher
PY - 2026
DA - 2026/09/07
JO - Open Research Europe
DO - 10.12688/openreseurope.24615.1
UR - https://hal.science/hal-05751701v1
AB - Background: Open science practices are increasingly promoted to improve the transparency and reproducibility of biomedical research, and their large-scale monitoring has been identified as a priority by initiatives such as the UNESCO Recommendation on Open Science. Existing automated tools cover only narrow subsets of practices. We developed and validated an automated tool, based on locally run open-weights large language models (LLMs), to extract a core set of 13 open science practices from published medical research articles. Methods: We built a validation database of 303 research articles published in 2020–2021 in 10 major general medical journals, stratified across randomized controlled trials (RCTs), meta-analyses, and other designs. Duplicate manual extraction with expert adjudication served as the reference standard. The pipeline combined document conversion (GROBID), BM25-based chunk retrieval, and classification by three Llama models (3-8B, 3-70B, 3.3-70B) run entirely locally, with and without supplementary materials. Diagnostic performance (sensitivity, specificity, predictive values, F1) was computed against the reference standard. Results: Usable outputs were obtained for 301/303 articles. Performance was heterogeneous across criteria rather than across models: practices reported in standardized statements (conflicts of interest, funding, author contributions, data sharing, pre-registration, reporting guidelines) were extracted reliably (F1 ≥ 0.75 for at least one model), whereas rare or inconsistently reported practices (code sharing, statistical analysis plan, protocol sharing, ORCID, open access status, preprints) remained difficult regardless of model size. Study-type classification exceeded 0.90 on all metrics. Larger models improved several criteria at a 6.6–7.5× computational cost, and adding supplementary materials did not improve performance, mainly due to conversion failures. Conclusions: A fully local, open-weights LLM pipeline can reliably monitor a substantial subset of open science practices. Document conversion and layout heterogeneity, rather than language understanding, constitute the main barrier to scaling automated open science monitoring.
SN - 2732-5121
ER -