Operationalising LLM-assisted screening of literature to support systematic reviews
- 1CSIRO Environment; Centre for Marine Socioecology, University of Tasmania
Abstract
Large language models (LLMs) can ease the work of screening titles and abstracts for systematic reviews, but obtaining reliable results requires researchers to make practical choices about which LLMs to use, how to combine their scores into a ranking, and how far down that ranking to read. We aimed to identify a general- purpose workflow that screens accurately, minimises human review effort, and generalises across environmental literature corpora. We ran an ensemble of five open-source LLMs across ten human-annotated systematic reviews from the field of ecology and environmental science spanning 19,155 studies. We then asked: (1) how well an ensemble of LLMs ranks relevant papers above irrelevant ones, and (2) where a human reviewer should stop working down that ranked list. A four-LLM ensemble chosen without any labels came close, on every review, to the best ranking achievable with that review's annotations (mean Average Precision 0.66 versus 0.68). We tested different rules for when to stop human review, finding that a single SAFE stopping rule chosen in advance recovered greater than 95% of relevant records on all ten reviews while requiring a human to screen 56% of the corpus on average. Against the established open-source active-learning tool ASReview, this label-free ranking reached the same recall target at lower human workload on seven of the ten reviews (39% versus 42% of the corpus on average). The paper offers a complete workflow that can be adopted for new, unlabelled reviews, using open-source LLMs small enough to run on a high-end consumer laptop, and we provide it as an open-source R package.
Related
- Operationalising the context in regenerative agriculture: decision-making and farm variability
- AOPTK: Open-Source Software for AI-Assisted Toxicological Literature Analysis
- The Functional Neurological Disorder, Volume 1: Non-systematic review from the scientific literature
- Serum Cystatin 4 combined with routine clinical laboratory indicators: A support vector machine diagnostic model for early screening of gastrointestinal tumors
- Definition of mild to moderate ulcerative colitis in clinical trials: A systematic literature review
BibTeX
@misc{spillias2026operationalising,
title = {{Operationalising LLM-assisted screening of literature to support systematic reviews}},
author = {S. Spillias and L. Avila-Turriago and C. Brown and A. Easton and J. Roberts and M. Sievers and S. Swearer and A. Taylor and B. Wright and V. Komyakova},
howpublished = {bioRxiv},
year = {2026},
month = jul,
doi = {10.64898/2026.07.29.741127},
url = {https://www.biorxiv.org/content/10.64898/2026.07.29.741127v2},
note = {CC BY-NC},
}
RIS
TY - UNPB
TI - Operationalising LLM-assisted screening of literature to support systematic reviews
AU - S. Spillias
AU - L. Avila-Turriago
AU - C. Brown
AU - A. Easton
AU - J. Roberts
AU - M. Sievers
AU - S. Swearer
AU - A. Taylor
AU - B. Wright
AU - V. Komyakova
PY - 2026
DA - 2026/07/29
PB - bioRxiv
DO - 10.64898/2026.07.29.741127
UR - https://www.biorxiv.org/content/10.64898/2026.07.29.741127v2
AB - Large language models (LLMs) can ease the work of screening titles and abstracts for systematic reviews, but obtaining reliable results requires researchers to make practical choices about which LLMs to use, how to combine their scores into a ranking, and how far down that ranking to read. We aimed to identify a general- purpose workflow that screens accurately, minimises human review effort, and generalises across environmental literature corpora. We ran an ensemble of five open-source LLMs across ten human-annotated systematic reviews from the field of ecology and environmental science spanning 19,155 studies. We then asked: (1) how well an ensemble of LLMs ranks relevant papers above irrelevant ones, and (2) where a human reviewer should stop working down that ranked list. A four-LLM ensemble chosen without any labels came close, on every review, to the best ranking achievable with that review's annotations (mean Average Precision 0.66 versus 0.68). We tested different rules for when to stop human review, finding that a single SAFE stopping rule chosen in advance recovered greater than 95% of relevant records on all ten reviews while requiring a human to screen 56% of the corpus on average. Against the established open-source active-learning tool ASReview, this label-free ranking reached the same recall target at lower human workload on seven of the ten reviews (39% versus 42% of the corpus on average). The paper offers a complete workflow that can be adopted for new, unlabelled reviews, using open-source LLMs small enough to run on a high-end consumer laptop, and we provide it as an open-source R package.
ER -