An auditable evidence compiler for large language model-assisted systematic reviews
- 1Department of Orthopaedics, Shanghai Ninth People's Hospital, Shanghai Jiao Tong University School of Medicine, Shanghai, China
Abstract
BackgroundErrors in interpreting sources, identifying overlapping cohorts and handling corrections can change the results of systematic reviews. We investigated how documented evidence-processing errors affected one completed review assisted by large language models (LLMs), and how an auditable workflow linked their detection and correction to released outputs.
MethodsWe conducted a retrospective, non-blinded methodological case study of a registered prognostic review. Frozen correction and audit records were reconciled into distinct root-cause events. We described their recorded consequences for analysis membership, study weights, pooled results, certainty and publication files, and traced four illustrative error pathways to implemented controls. Six release-integrity domains provided supporting internal checks. The production review was also the development case; there was no external comparator or independent human reference.
ResultsFifty root-cause events were documented; 12 had changed pooled results before correction. Forty-six were resolved and four remained disclosed non-critical limitations. Twenty-one events had at least one recorded scientific consequence. The four pathways concerned survival time origin, cohort dependence, outcome direction and certainty assessment. Corrections changed analysis membership, weighting or interpretation; in the certainty case, the risk-of-bias domain changed while the final certainty grade remained at its lowest level. The application comprised 454 reports and 445 studies, with 41 of 49 registered analyses fitted. Public records link the reported defects, corrections, rebuilds and release checks.
ConclusionsIn this review, consequential errors arose in the relationships between source judgments and their downstream use. Explicit estimand rules, adjudication-driven dependence mapping, source-based outcome coding and documented downstream rebuilds made corrections inspectable. The case provides operational rules and documented failure examples for other review teams to examine and test; it does not establish error rates or comparative effectiveness.
Systematic review registrationUnderlying clinical review: PROSPERO CRD420261449181. The retrospective methods evaluation was not prospectively registered.
Related
- Prospective Clinical Evidence on Artificial Intelligence-Enabled Augmented and Mixed Reality in Robot-Assisted Prostate and Kidney Cancer Surgery: A Systematic Review.
- Transparent Weighting of Heterogeneous Evidence for Auditable Candidate Prioritization
- Deploying YOLO for whole-slide mitosis detection: A transparent, auditable, and reproducible pipeline for pathology AI-assisted diagnosis and decision-support tool
- Effect of guideline-assisted prompting on large language model performance in medication-related osteonecrosis of the jaw–related endodontic decision-making
- Language model-assisted label refinement for accurate sepsis detection from electronic health records
BibTeX
@misc{yin2026auditable,
title = {{An auditable evidence compiler for large language model-assisted systematic reviews}},
author = {C. Yin and Z. Jing and Z. Zhang},
howpublished = {medRxiv},
year = {2026},
month = sep,
doi = {10.64898/2026.09.21.26363538},
url = {https://www.medrxiv.org/content/10.64898/2026.09.21.26363538v1},
note = {CC BY},
}
RIS
TY - UNPB
TI - An auditable evidence compiler for large language model-assisted systematic reviews
AU - C. Yin
AU - Z. Jing
AU - Z. Zhang
PY - 2026
DA - 2026/09/21
PB - medRxiv
DO - 10.64898/2026.09.21.26363538
UR - https://www.medrxiv.org/content/10.64898/2026.09.21.26363538v1
AB - BackgroundErrors in interpreting sources, identifying overlapping cohorts and handling corrections can change the results of systematic reviews. We investigated how documented evidence-processing errors affected one completed review assisted by large language models (LLMs), and how an auditable workflow linked their detection and correction to released outputs. MethodsWe conducted a retrospective, non-blinded methodological case study of a registered prognostic review. Frozen correction and audit records were reconciled into distinct root-cause events. We described their recorded consequences for analysis membership, study weights, pooled results, certainty and publication files, and traced four illustrative error pathways to implemented controls. Six release-integrity domains provided supporting internal checks. The production review was also the development case; there was no external comparator or independent human reference. ResultsFifty root-cause events were documented; 12 had changed pooled results before correction. Forty-six were resolved and four remained disclosed non-critical limitations. Twenty-one events had at least one recorded scientific consequence. The four pathways concerned survival time origin, cohort dependence, outcome direction and certainty assessment. Corrections changed analysis membership, weighting or interpretation; in the certainty case, the risk-of-bias domain changed while the final certainty grade remained at its lowest level. The application comprised 454 reports and 445 studies, with 41 of 49 registered analyses fitted. Public records link the reported defects, corrections, rebuilds and release checks. ConclusionsIn this review, consequential errors arose in the relationships between source judgments and their downstream use. Explicit estimand rules, adjudication-driven dependence mapping, source-based outcome coding and documented downstream rebuilds made corrections inspectable. The case provides operational rules and documented failure examples for other review teams to examine and test; it does not establish error rates or comparative effectiveness. Systematic review registrationUnderlying clinical review: PROSPERO CRD420261449181. The retrospective methods evaluation was not prospectively registered.
ER -