RoB2-Evaluator: Development and Technical Evaluation of a Rule-Constrained Large Language Model System for Cochrane Risk of Bias 2 Assessment
- 1University of Toronto
Abstract
BackgroundCochrane Risk of Bias 2 (RoB 2) assessment is methodologically demanding and resource intensive. Large language models may assist this process but can produce variable judgments.
AimTo develop a rule-constrained LLM system for RoB 2 assessment and evaluate its temporal stability and agreement with human reviewers.
MethodsRoB2-Evaluator was developed using GPT-5.2 with fixed RoB 2 guidance and a locked instruction framework. Seven RCTs were used for calibration before version freezing. The frozen system was evaluated on 10 separate RCTs, with repeat assessment after one week and comparison against consensus judgments from two independent human reviewers.
ResultsTemporal agreement was 90.0% (weighted {kappa} = 0.93), and agreement with human consensus was also 90.0% (weighted {kappa} = 0.93). Methodological rationales were stable in 94.3% of repeated comparisons. Full AI-human conceptual alignment occurred in 85.7%.
ConclusionRoB2-Evaluator showed encouraging post-freeze stability, human agreement, and rationale consistency, supporting larger independent evaluations.
Related
- Development of a Dedicated Prostate Phantom for Technical Evaluation of Diffusion-weighted MR Imaging Sequences
- Technical Feasibility of a Multimodal Electro-Optical and Infrared Imaging Pipeline for Postoperative Wound Assessment After Total Knee Arthroplasty: Formative Evaluation Study.
- Development and quality evaluation of lyophilized plasma external quality assessment samples for coagulation testing in resource-limited settings
- Development and Population-Based Prognostic Utility Assessment of an ICD-10-CM-Based Prediction Model to Identify Left Ventricular Systolic Dysfunction: Discrimination Performance and Population-Based Evaluation in 83,357 Patients with Acute Coronary Syndrome
- Evaluator-Masked Assessment of 4 Consumer-Facing Large Language Models on Expert Anterior Segment Ophthalmology Questions.
BibTeX
@misc{joseph2026rob2evaluator,
title = {{RoB2-Evaluator: Development and Technical Evaluation of a Rule-Constrained Large Language Model System for Cochrane Risk of Bias 2 Assessment}},
author = {J. T. Joseph and R. Vishwanath and J. V. Sisirkumar},
howpublished = {medRxiv},
year = {2026},
month = sep,
doi = {10.64898/2026.09.22.26363693},
url = {https://www.medrxiv.org/content/10.64898/2026.09.22.26363693v1},
note = {CC BY},
}
RIS
TY - UNPB
TI - RoB2-Evaluator: Development and Technical Evaluation of a Rule-Constrained Large Language Model System for Cochrane Risk of Bias 2 Assessment
AU - J. T. Joseph
AU - R. Vishwanath
AU - J. V. Sisirkumar
PY - 2026
DA - 2026/09/22
PB - medRxiv
DO - 10.64898/2026.09.22.26363693
UR - https://www.medrxiv.org/content/10.64898/2026.09.22.26363693v1
AB - BackgroundCochrane Risk of Bias 2 (RoB 2) assessment is methodologically demanding and resource intensive. Large language models may assist this process but can produce variable judgments. AimTo develop a rule-constrained LLM system for RoB 2 assessment and evaluate its temporal stability and agreement with human reviewers. MethodsRoB2-Evaluator was developed using GPT-5.2 with fixed RoB 2 guidance and a locked instruction framework. Seven RCTs were used for calibration before version freezing. The frozen system was evaluated on 10 separate RCTs, with repeat assessment after one week and comparison against consensus judgments from two independent human reviewers. ResultsTemporal agreement was 90.0% (weighted {kappa} = 0.93), and agreement with human consensus was also 90.0% (weighted {kappa} = 0.93). Methodological rationales were stable in 94.3% of repeated comparisons. Full AI-human conceptual alignment occurred in 85.7%. ConclusionRoB2-Evaluator showed encouraging post-freeze stability, human agreement, and rationale consistency, supporting larger independent evaluations.
ER -