AI as gatekeeper: a cross-sectional study of AI-generated peer review of landmark oncological studies
- 1Department of Medical Oncology, Gulhane Training and Research Hospital, Ankara, Türkiye
- 2Department of Medical Oncology, Ankara University, Ankara, Türkiye
Abstract
Background: The increasing volume of scientific manuscripts has strained traditional peer-review systems, prompting interest in artificial intelligence (AI) as a potential solution. However, the comparative performance of large language models (LLMs) in critically appraising oncological research remains unexplored.
Methods: We conducted a cross-sectional study evaluating the peer-review capabilities of three LLMs—Claude Sonnet 4.5, Google Gemini 3.0 Thinking, and OpenAI ChatGPT 5.1—using 50 blinded landmark oncology trials identified from ASCO, ESMO, and NCCN guidelines. Manuscripts were anonymized by removing identifying information and submitted to each model with a standardized prompt. Two investigators independently analyzed reviews for final verdicts, trial awareness, journal suitability assessments, and concerns across five domains (biological, methodological, statistical, trial groups, and reporting). Chi-square tests with Bonferroni correction and logistic regression were used for statistical analysis.
Results: Significant differences were observed in reviewer verdicts among models (p < 0.001). Claude Sonnet 4.5 was most stringent (74% major revisions, 6% rejections), while Google Gemini 3.0 was most lenient (58% minor revisions, 4% acceptances). Trial awareness varied significantly (Claude 54%, Gemini 50%, ChatGPT 14%; p < 0.001). Methodological concerns differed among models (Claude 86%, ChatGPT 80%, Gemini 62%; p = 0.014). Logistic regression identified statistical concerns as the only significant predictor of critical verdicts (OR = 8.57; 95% CI 1.28–57.4; p = 0.027). Journal suitability was assessed by Claude in 90% of reviews but by Gemini in none.
Conclusion: LLMs demonstrate heterogeneous performance in peer-reviewing landmark oncology trials, with significant variation in stringency, trial recognition, and methodological scrutiny. These findings highlight the need for standardized frameworks before integrating AI into peer-review processes.
Supplementary Information: The online version contains supplementary material available at https://doi.org/10.1186/s12874-026-02978-y.
Related
- Patient-Perceived and Record-Based Concordance of AI-Generated Perioperative Information After Total Hip Arthroplasty: A Cross-Sectional Study with Multidisciplinary Review
- N-Glycosylation in the Receptor Binding Domain serves as aggregation gatekeeper of the SARS CoV2 Spike glycoprotein folding
- When expertise fails: detection of artificial intelligence - generated vitiligo imagery across the spectrum of dermatological experience; a cross-sectional exploratory study
- Using AI-generated ultra-widefield-like fundus images to assist colour fundus photography in screening for chronic kidney disease: a multicentre cross-sectional study
- AI-generated high-intensity interval training information: a cross-sectional comparative evaluation of quality and readability
BibTeX
@article{erdat2026gatekeeper,
title = {{AI as gatekeeper: a cross-sectional study of AI-generated peer review of landmark oncological studies}},
author = {Efe Cem Erdat and Güngör Utkan},
journal = {BMC Medical Research Methodology},
year = {2026},
month = aug,
doi = {10.1186/s12874-026-02978-y},
url = {https://doi.org/10.1186/s12874-026-02978-y},
note = {CC BY-NC-ND 4.0},
}
RIS
TY - JOUR
TI - AI as gatekeeper: a cross-sectional study of AI-generated peer review of landmark oncological studies
AU - Efe Cem Erdat
AU - Güngör Utkan
PY - 2026
DA - 2026/08/18
JO - BMC Medical Research Methodology
DO - 10.1186/s12874-026-02978-y
UR - https://doi.org/10.1186/s12874-026-02978-y
AB - Background: The increasing volume of scientific manuscripts has strained traditional peer-review systems, prompting interest in artificial intelligence (AI) as a potential solution. However, the comparative performance of large language models (LLMs) in critically appraising oncological research remains unexplored. Methods: We conducted a cross-sectional study evaluating the peer-review capabilities of three LLMs—Claude Sonnet 4.5, Google Gemini 3.0 Thinking, and OpenAI ChatGPT 5.1—using 50 blinded landmark oncology trials identified from ASCO, ESMO, and NCCN guidelines. Manuscripts were anonymized by removing identifying information and submitted to each model with a standardized prompt. Two investigators independently analyzed reviews for final verdicts, trial awareness, journal suitability assessments, and concerns across five domains (biological, methodological, statistical, trial groups, and reporting). Chi-square tests with Bonferroni correction and logistic regression were used for statistical analysis. Results: Significant differences were observed in reviewer verdicts among models (p < 0.001). Claude Sonnet 4.5 was most stringent (74% major revisions, 6% rejections), while Google Gemini 3.0 was most lenient (58% minor revisions, 4% acceptances). Trial awareness varied significantly (Claude 54%, Gemini 50%, ChatGPT 14%; p < 0.001). Methodological concerns differed among models (Claude 86%, ChatGPT 80%, Gemini 62%; p = 0.014). Logistic regression identified statistical concerns as the only significant predictor of critical verdicts (OR = 8.57; 95% CI 1.28–57.4; p = 0.027). Journal suitability was assessed by Claude in 90% of reviews but by Gemini in none. Conclusion: LLMs demonstrate heterogeneous performance in peer-reviewing landmark oncology trials, with significant variation in stringency, trial recognition, and methodological scrutiny. These findings highlight the need for standardized frameworks before integrating AI into peer-review processes. Supplementary Information: The online version contains supplementary material available at https://doi.org/10.1186/s12874-026-02978-y.
SN - 1471-2288
ER -