Auditing Biomedical Literature with Blinded AI Peer Review: Where Peer Review Works, Where It Fails, and How to Strengthen It
- 1Institute for Genomics and Evolutionary Medicine, Department of Biology, Temple University, Philadelphia, PA, USA
Abstract
Biomedical publishing has grown more than elevenfold since 1991, outstripping human peer review. Unblinded language models cannot audit this literature objectively because they recognize famous authors and journals from training memory. We built BARS (Blinded AI Review System), an automated referee that strips titles, author names, institutions, journal headers, and dates before grading full-text manuscripts on a 10-point scale covering methods, evidence, biology, and reproducibility. We used BARS to generate 1,874 blind reviews across 1,714 unique papers. BARS passed 100% of verified sound controls, caught 77% to 91% of internally flawed papers, eliminated a +0.35-point prestige boost, and more than doubled human referee consistency. Across 320 recent open-access articles (2020-2025) from eight fields, 54% failed on core scientific rigor (<5.0/10; 41% excluding megajournals). Revision at selective journals improved preprints more than twice as much as revision at commercial megajournals (+0.62 vs. +0.30 points; baseline-adjusted p = 0.033), and BARS recalled 98% of unresolved human referee critiques across 100 open-review papers. Across 915 date-blinded papers from 1991 to 2025, average scores fell from 7.6/10 to 5.8/10 as megajournals and template database studies expanded. Blinded self-audits of 238 papers by two senior authors -- and of this manuscript itself -- caught real statistical, control, and table errors in our own work. At four cents per paper, BARS offers a practical quality check for authors, journals, and readers (https://veg.github.io/BARS/).
Related
- From Tool State to Scientific Action: Auditing Decision Provenance in Biomedical Research Agents
- DBCLS BioHackathon 2026 report: Open, Reproducible Extraction of Data-Entity References from the Biomedical Literature – with Entry-Level Resolution via Identifiers.org / TogoID
- Strategic communication to strengthen nurses’ professional identity and drive cultural change in healthcare organizations: A JBI scoping review protocol
- Low statistical power in biomedical science: a review of three human research domains
- Female Monarch Butterflies Are More Successful than Males During the Fall Migration: A Literature Review
BibTeX
@misc{pond2026auditing,
title = {{Auditing Biomedical Literature with Blinded AI Peer Review: Where Peer Review Works, Where It Fails, and How to Strengthen It}},
author = {S. L. Kosakovsky Pond and S. J. K. Pond and S. Weaver and J. D. Zehr and D. P. Martin and A. Nekrutenko},
howpublished = {bioRxiv},
year = {2026},
month = nov,
doi = {10.64898/2026.11.02.763753},
url = {https://www.biorxiv.org/content/10.64898/2026.11.02.763753v1},
note = {CC BY},
}
RIS
TY - UNPB
TI - Auditing Biomedical Literature with Blinded AI Peer Review: Where Peer Review Works, Where It Fails, and How to Strengthen It
AU - S. L. Kosakovsky Pond
AU - S. J. K. Pond
AU - S. Weaver
AU - J. D. Zehr
AU - D. P. Martin
AU - A. Nekrutenko
PY - 2026
DA - 2026/11/02
PB - bioRxiv
DO - 10.64898/2026.11.02.763753
UR - https://www.biorxiv.org/content/10.64898/2026.11.02.763753v1
AB - Biomedical publishing has grown more than elevenfold since 1991, outstripping human peer review. Unblinded language models cannot audit this literature objectively because they recognize famous authors and journals from training memory. We built BARS (Blinded AI Review System), an automated referee that strips titles, author names, institutions, journal headers, and dates before grading full-text manuscripts on a 10-point scale covering methods, evidence, biology, and reproducibility. We used BARS to generate 1,874 blind reviews across 1,714 unique papers. BARS passed 100% of verified sound controls, caught 77% to 91% of internally flawed papers, eliminated a +0.35-point prestige boost, and more than doubled human referee consistency. Across 320 recent open-access articles (2020-2025) from eight fields, 54% failed on core scientific rigor (<5.0/10; 41% excluding megajournals). Revision at selective journals improved preprints more than twice as much as revision at commercial megajournals (+0.62 vs. +0.30 points; baseline-adjusted p = 0.033), and BARS recalled 98% of unresolved human referee critiques across 100 open-review papers. Across 915 date-blinded papers from 1991 to 2025, average scores fell from 7.6/10 to 5.8/10 as megajournals and template database studies expanded. Blinded self-audits of 238 papers by two senior authors -- and of this manuscript itself -- caught real statistical, control, and table errors in our own work. At four cents per paper, BARS offers a practical quality check for authors, journals, and readers (https://veg.github.io/BARS/).
ER -