A large language model for risk-of-bias assessment in systematic reviews of prognosis studies in clinical neurology
- 1Department of Child Neurology, UMC Utrecht Brain Center, University Medical Center Utrecht and Utrecht University, Utrecht, The Netherlands, member of ERN EpiCA
Abstract
BackgroundRisk-of-bias (ROB) assessments represent an integral component of systematic reviews. However, this task is often highly repetitive, time-consuming, and may lack inter-rater consistency. Large language models (LLMs) offer opportunities for automation in systematic reviews, which may expedite and enhance the quality and consistency of research synthesis.
MethodsUsing zero-shot prompting, we designed an LLM-based pipeline as a virtual mimic of a human reviewer for the Quality in Prognosis Studies (QUIPS) framework. Then, focusing on prognostic research in a single discipline (neurology), we applied this pipeline to articles included in previously published systematic reviews. We studied inter-rater agreement between both (1) the LLM and the original human ROB assessments and (2) between original human ROB assessments.
Results298 articles from 15 reviews across three domains (epilepsy, traumatic brain injury, stroke) were included. We demonstrate the feasibility of a tailored, prompt-engineered LLM pipeline for automating ROB assessments with the QUIPS tool. While LLM-human agreement was limited (Cohens weighted kappa; = 0.22, 95% CI, 0.12 - 0.33), our data tentatively suggest, based on a small sample (n=5), that it may not be inferior to human-human agreement (Cohens weighted kappa; = -0.25, 95% CI, -1.04 - 0.54). Wilcoxon signed-rank tests were statistically significant (p < 0.05) across four bias domains and for the overall risk scores, and rank-biserial correlations demonstrated human raters tendency to assign higher risk scores than LLM counterparts.
ConclusionsWith targeted methodological refinements - including standardization of QUIPS implementation and validation against expert ratings - automated ROB assessments may meaningfully reduce time and cost of systematic reviews of prognosis studies in neurology and beyond.
Related
- The therapeutic landscape of cannabidiol: a systematic review of molecular mechanisms, clinical efficacy in oncology and neurology, and regulatory changes.
- Remote self-administered speech assessment in clinical neurology: feasibility, data quality, and equity in 1,000 consecutive patients
- Artificial Intelligence in Eye Disease: A Systematic Review of Screening, Diagnosis, Prognosis, Treatment-Response Prediction, and Clinical Decision Support
- Accuracy and confidence calibration of large language models on neurology board-style questions
- Integrated Molecular Classification of Endometrial Cancer: Impact on Diagnosis, Prognosis, and Clinical Management—A Systematic Review
BibTeX
@misc{kampman2026large,
title = {{A large language model for risk-of-bias assessment in systematic reviews of prognosis studies in clinical neurology}},
author = {S. L. Kampman and K. P. Braun and W. M. Otte},
howpublished = {medRxiv},
year = {2026},
month = sep,
doi = {10.64898/2026.09.09.26362643},
url = {https://www.medrxiv.org/content/10.64898/2026.09.09.26362643v1},
note = {CC BY},
}
RIS
TY - UNPB
TI - A large language model for risk-of-bias assessment in systematic reviews of prognosis studies in clinical neurology
AU - S. L. Kampman
AU - K. P. Braun
AU - W. M. Otte
PY - 2026
DA - 2026/09/09
PB - medRxiv
DO - 10.64898/2026.09.09.26362643
UR - https://www.medrxiv.org/content/10.64898/2026.09.09.26362643v1
AB - BackgroundRisk-of-bias (ROB) assessments represent an integral component of systematic reviews. However, this task is often highly repetitive, time-consuming, and may lack inter-rater consistency. Large language models (LLMs) offer opportunities for automation in systematic reviews, which may expedite and enhance the quality and consistency of research synthesis. MethodsUsing zero-shot prompting, we designed an LLM-based pipeline as a virtual mimic of a human reviewer for the Quality in Prognosis Studies (QUIPS) framework. Then, focusing on prognostic research in a single discipline (neurology), we applied this pipeline to articles included in previously published systematic reviews. We studied inter-rater agreement between both (1) the LLM and the original human ROB assessments and (2) between original human ROB assessments. Results298 articles from 15 reviews across three domains (epilepsy, traumatic brain injury, stroke) were included. We demonstrate the feasibility of a tailored, prompt-engineered LLM pipeline for automating ROB assessments with the QUIPS tool. While LLM-human agreement was limited (Cohens weighted kappa; = 0.22, 95% CI, 0.12 - 0.33), our data tentatively suggest, based on a small sample (n=5), that it may not be inferior to human-human agreement (Cohens weighted kappa; = -0.25, 95% CI, -1.04 - 0.54). Wilcoxon signed-rank tests were statistically significant (p < 0.05) across four bias domains and for the overall risk scores, and rank-biserial correlations demonstrated human raters tendency to assign higher risk scores than LLM counterparts. ConclusionsWith targeted methodological refinements - including standardization of QUIPS implementation and validation against expert ratings - automated ROB assessments may meaningfully reduce time and cost of systematic reviews of prognosis studies in neurology and beyond.
ER -