Title and abstract screening for systematic reviews with Jev, a System One model: comparison with generative large language models
- 1National Center of Neurology and Psychiatry
Abstract
Large language models (LLMs) screen titles and abstracts without review-specific training, but generating screening decisions as text takes processing time and incurs API charges. We evaluated Jev, a non-generative model returning classification probabilities, on 4527 records from two systematic reviews of bipolar disorder treatments. We used human reviewers decisions to retain records for full-text assessment as the reference standard. In the primary analysis, we asked Jev yes-or-no questions ("Noul") about whether to retain each record. Records were retained when their retention probability was at least 50%, a cutoff specified before the final evaluation runs. We also evaluated lower retention thresholds. For comparison, we gave Jev explicit include/exclude options ("Choice") and used the same prompts with GPT-5 mini and GPT-6 Astra; both LLMs used default reasoning. In the primary analysis at 50%, sensitivity and specificity were 99.6% and 98.9% for light therapy and 93.2% and 93.5% for adjunctive pharmacotherapy. Lower cutoffs selected on these data retained all reference-positive records. Compared with assessing every record manually, these cutoffs reduced the number requiring human assessment by 96.9% for light therapy and 61.3% for adjunctive pharmacotherapy. For light therapy and adjunctive pharmacotherapy, sensitivity was 92.3% and 90.0% with Jev using explicit options at 50%, 87.2% and 90.0% with GPT-5 mini, and 92.3% and 80.8% with GPT-6 Astra. Costs for screening all 4527 records once were US$0.20 for Jev using explicit options, US$2.59 for GPT-5 mini, and US$33.45 for GPT-6 Astra. Jevs probabilities could provide graded information for reviewers assessing titles and abstracts.
HighlightsO_ST_ABSWhat is already knownC_ST_ABSGenerative LLMs can screen titles and abstracts using natural-language eligibility criteria, but generating each decision takes time and incurs API charges.
What is newJev screened records without generating text or requiring training on records labeled for each review. In two reviews, screening cost substantially less than with the tested LLM configurations.
Potential impact for Research Synthesis Methods readersRather than providing only an include-or-exclude decision, Jev indicates how strongly the model favors retaining each record. Reviewers could consider this information alongside the title, abstract, and eligibility criteria.
Related
- Harnessing conversational generative artificial intelligence and natural language processing to address the colorectal cancer screening adherence gap: the Azores endeavour
- Generative Language Models for Disease Treatment Recommendations: A Systematic Literature Review
- Knowledge distillation-based lightweight generative large language models for standardized quality control of Chinese radiology reports
- Automated language impairment screening in acute stroke using connected speech
- Comparison of two automated PGMI-based software tools for breast positioning quality assessment in DBT screening
BibTeX
@misc{matsui2026title,
title = {{Title and abstract screening for systematic reviews with Jev, a System One model: comparison with generative large language models}},
author = {K. Matsui and Y. Takaesu},
howpublished = {medRxiv},
year = {2026},
month = sep,
doi = {10.64898/2026.09.25.26364021},
url = {https://www.medrxiv.org/content/10.64898/2026.09.25.26364021v1},
note = {CC BY},
}
RIS
TY - UNPB
TI - Title and abstract screening for systematic reviews with Jev, a System One model: comparison with generative large language models
AU - K. Matsui
AU - Y. Takaesu
PY - 2026
DA - 2026/09/25
PB - medRxiv
DO - 10.64898/2026.09.25.26364021
UR - https://www.medrxiv.org/content/10.64898/2026.09.25.26364021v1
AB - Large language models (LLMs) screen titles and abstracts without review-specific training, but generating screening decisions as text takes processing time and incurs API charges. We evaluated Jev, a non-generative model returning classification probabilities, on 4527 records from two systematic reviews of bipolar disorder treatments. We used human reviewers decisions to retain records for full-text assessment as the reference standard. In the primary analysis, we asked Jev yes-or-no questions ("Noul") about whether to retain each record. Records were retained when their retention probability was at least 50%, a cutoff specified before the final evaluation runs. We also evaluated lower retention thresholds. For comparison, we gave Jev explicit include/exclude options ("Choice") and used the same prompts with GPT-5 mini and GPT-6 Astra; both LLMs used default reasoning. In the primary analysis at 50%, sensitivity and specificity were 99.6% and 98.9% for light therapy and 93.2% and 93.5% for adjunctive pharmacotherapy. Lower cutoffs selected on these data retained all reference-positive records. Compared with assessing every record manually, these cutoffs reduced the number requiring human assessment by 96.9% for light therapy and 61.3% for adjunctive pharmacotherapy. For light therapy and adjunctive pharmacotherapy, sensitivity was 92.3% and 90.0% with Jev using explicit options at 50%, 87.2% and 90.0% with GPT-5 mini, and 92.3% and 80.8% with GPT-6 Astra. Costs for screening all 4527 records once were US$0.20 for Jev using explicit options, US$2.59 for GPT-5 mini, and US$33.45 for GPT-6 Astra. Jevs probabilities could provide graded information for reviewers assessing titles and abstracts. HighlightsO_ST_ABSWhat is already knownC_ST_ABSGenerative LLMs can screen titles and abstracts using natural-language eligibility criteria, but generating each decision takes time and incurs API charges. What is newJev screened records without generating text or requiring training on records labeled for each review. In two reviews, screening cost substantially less than with the tested LLM configurations. Potential impact for Research Synthesis Methods readersRather than providing only an include-or-exclude decision, Jev indicates how strongly the model favors retaining each record. Reviewers could consider this information alongside the title, abstract, and eligibility criteria.
ER -