Evaluation of Elicit as an adjunct search, screening, and data extraction tool for systematic reviews
- 1Centre for Addiction and Mental Health
Abstract
BackgroundSystematic reviews are critical for informing evidence-based medicine, policy, and future research. However, conducting a systematic review is labour- and cost-intensive and error-prone. Elicit is an AI tool that claims to replicate the systematic review process.
MethodsOur study investigated Elicits ability to replicate a systematic review by comparing its results with those of an existing systematic review recently published by our research group. We calculated accuracy, precision, sensitivity, error rate, and person-time savings at each stage of the systematic review.
ResultsAt search, Elicit had poor overlap, finding 158 of the 5285 studies. For abstract and full-text screening, Elicit had moderate sensitivity and accuracy but poor precision, with a skew toward false positives. Elicits data extraction tool was promising, with sentence-specific citations, and provided helpful information for 85% of values even when it did not find exact matches. Elicit offered person-time savings, taking 3.16 hours to complete the full systematic review process.
ConclusionsElicit shows promise as an adjunct screening and extraction AI tool, but in its current form, it is not ready as a standalone tool to replace the systematic review process.
Related
- CHARMS and PROBAST+AI: an updated template for Data Extraction and Risk of Bias Assessment in systematic reviews of prediction models
- Title and abstract screening for systematic reviews with Jev, a System One model: comparison with generative large language models
- Validation of a two-stage automated screening pipeline for medical systematic reviews: a stratified concordance study with three independent human reviewers, using PABAK and Gwet's AC1
- Deploying Local Large Language Models for Automated Article Screening in Scientific Literature Reviews
- School-based mindfulness interventions in children and adolescents: an umbrella review of systematic reviews, with implications for physical education
BibTeX
@misc{wei2026evaluation,
title = {{Evaluation of Elicit as an adjunct search, screening, and data extraction tool for systematic reviews}},
author = {I. Wei and J. N. Shah and A. Miah and C. Hawco and F. Coutts and L. Sikstrom and S. H. Ameis and A. N. Voineskos and T. Glatard and E. W. Dickie},
howpublished = {medRxiv},
year = {2026},
month = oct,
doi = {10.64898/2026.10.02.26364621},
url = {https://www.medrxiv.org/content/10.64898/2026.10.02.26364621v1},
note = {CC BY},
}
RIS
TY - UNPB
TI - Evaluation of Elicit as an adjunct search, screening, and data extraction tool for systematic reviews
AU - I. Wei
AU - J. N. Shah
AU - A. Miah
AU - C. Hawco
AU - F. Coutts
AU - L. Sikstrom
AU - S. H. Ameis
AU - A. N. Voineskos
AU - T. Glatard
AU - E. W. Dickie
PY - 2026
DA - 2026/10/02
PB - medRxiv
DO - 10.64898/2026.10.02.26364621
UR - https://www.medrxiv.org/content/10.64898/2026.10.02.26364621v1
AB - BackgroundSystematic reviews are critical for informing evidence-based medicine, policy, and future research. However, conducting a systematic review is labour- and cost-intensive and error-prone. Elicit is an AI tool that claims to replicate the systematic review process. MethodsOur study investigated Elicits ability to replicate a systematic review by comparing its results with those of an existing systematic review recently published by our research group. We calculated accuracy, precision, sensitivity, error rate, and person-time savings at each stage of the systematic review. ResultsAt search, Elicit had poor overlap, finding 158 of the 5285 studies. For abstract and full-text screening, Elicit had moderate sensitivity and accuracy but poor precision, with a skew toward false positives. Elicits data extraction tool was promising, with sentence-specific citations, and provided helpful information for 85% of values even when it did not find exact matches. Elicit offered person-time savings, taking 3.16 hours to complete the full systematic review process. ConclusionsElicit shows promise as an adjunct screening and extraction AI tool, but in its current form, it is not ready as a standalone tool to replace the systematic review process.
ER -