Methodological Adherence, Reporting Completeness, and Risk of Bias in AI-Assisted Systematic Reviews: A Matched Meta-Research Study
- 1University of Toronto
Abstract
Background: Artificial intelligence (AI) is increasingly used to accelerate systematic reviews that inform clinical practice and policy. However, evidence on the methodological adherence, reporting completeness, and risk of bias of completed AI-assisted reviews is limited. Objective: To compare methodological adherence, reporting completeness, and risk of bias between published systematic reviews with reported AI assistance and matched reviews with no reported AI use, and to assess AI-use transparency. Methods: We conducted a matched-cohort meta-research study of English-language, peer-reviewed health systematic reviews. AI-assisted reviews published during 2023--2025 were matched 1:2 to reviews with no reported AI use using individualized MEDLINE searches targeting publication timing, clinical domain, review type, and meta-analysis status. When more than two eligible comparators were available, two were randomly selected. AI assistance was defined as reported use during searching, screening, data extraction, risk-of-bias assessment, evidence synthesis, or substantive drafting. Full texts and supplementary materials were examined to classify reported AI use. The primary outcome was a prespecified modified 13-item AMSTAR-2 methodological-adherence score (range 0 -- 13; noninferiority margin &-1 point). Secondary outcomes included PRISMA 2020 reporting completeness and ROBIS risk of bias; AI-use transparency was assessed descriptively. Adapted 13-item AMSTAR-2 confidence categories were summarized descriptively, while critical- and noncritical-domain adherence was examined in exploratory adjusted analyses. Multivariable regression models adjusted for journal impact tier, funding reporting, formal review-group affiliation, review-team size, and review-question type, using cluster-robust standard errors by matched set. Sensitivity analyses included matched-set fixed-effects models. The study was registered on the Open Science Framework (DOI: 10.17605/OSF.IO/Q5JRW). Results: The cohort included 231 systematic reviews: 77 AI-assisted and 154 with no reported AI use. Mean 13-item AMSTAR-2 scores were 7.58 and 9.63, respectively. In multivariable-adjusted analyses, AI-assisted reviews scored 1.64 points lower (95% CI &-2.37 to &- 0.91); noninferiority was not demonstrated. PRISMA adherence was estimated to be 4.7 percentage points lower among AI-assisted reviews in primary adjusted analyses (95% CI &-8.9 to &-0.6) and 6.4 points lower in matched-set fixed-effects analyses (95% CI &-10.7 to &-2.1). Among reviews with assessable ROBIS ratings, 75.0% of AI-assisted reviews and 65.6% of comparators were rated low risk (adjusted difference +17.8 percentage points; 95% CI 2.7 to 33.0). Among AI-assisted reviews, model version or date was reported in 42.9%, prompts or parameters in 55.8%, and data-governance or ethics considerations in 9.1%. Conclusions: Reviews reporting AI assistance had lower methodological-adherence scores and had less complete reporting, despite more favorable risk-of-bias ratings. These findings provide an early warning that AI-assisted systematic reviews may require stronger safeguards, human verification, and transparency to support trust.
Related
- Methodological drivers, reporting completeness, and transferability in HPV vaccination economic evaluations: A scoping review
- Exercise for tension-type headache: protocol for a systematic review and meta-analysis with meta-regression and assessment of exercise reporting completeness.
- Reporting Completeness of Biliary Atresia Organoid Studies: A Scoping Review and Structured Audit
- A non-destructive methodological framework for modernizing legacy clinical reporting systems for AI-driven pharmacoinformatics: a SAS case study
- Lung-Reporting and Data System Category (Lung-RADS) and Adherence to Low-Dose Computed Tomography Screening and Recommended Follow-Up: A Multicenter Cohort Study
BibTeX
@misc{jay2026methodological,
title = {{Methodological Adherence, Reporting Completeness, and Risk of Bias in AI-Assisted Systematic Reviews: A Matched Meta-Research Study}},
author = {M. Jay and M. Esttaifo and A. Samani and E. Wilson and S. Esttaifo and M. Morgan and J. Abi-Jaoude and G. Reszetnik and M. Chopra and A. Alrashidi and J. K. Monteiro and G. Schemitsch and V. Karthikeyan and O. Obi and F. Emdin and D.-D. Nguyen and N. S. Anni and R. Sutradhar and C. Dharma and C. Yu and S. Straus and L. L. Lipscombe and A. Eskander},
howpublished = {medRxiv},
year = {2026},
month = nov,
doi = {10.64898/2026.11.06.26367388},
url = {https://www.medrxiv.org/content/10.64898/2026.11.06.26367388v1},
note = {CC BY-NC-ND},
}
RIS
TY - UNPB
TI - Methodological Adherence, Reporting Completeness, and Risk of Bias in AI-Assisted Systematic Reviews: A Matched Meta-Research Study
AU - M. Jay
AU - M. Esttaifo
AU - A. Samani
AU - E. Wilson
AU - S. Esttaifo
AU - M. Morgan
AU - J. Abi-Jaoude
AU - G. Reszetnik
AU - M. Chopra
AU - A. Alrashidi
AU - J. K. Monteiro
AU - G. Schemitsch
AU - V. Karthikeyan
AU - O. Obi
AU - F. Emdin
AU - D.-D. Nguyen
AU - N. S. Anni
AU - R. Sutradhar
AU - C. Dharma
AU - C. Yu
AU - S. Straus
AU - L. L. Lipscombe
AU - A. Eskander
PY - 2026
DA - 2026/11/06
PB - medRxiv
DO - 10.64898/2026.11.06.26367388
UR - https://www.medrxiv.org/content/10.64898/2026.11.06.26367388v1
AB - Background: Artificial intelligence (AI) is increasingly used to accelerate systematic reviews that inform clinical practice and policy. However, evidence on the methodological adherence, reporting completeness, and risk of bias of completed AI-assisted reviews is limited. Objective: To compare methodological adherence, reporting completeness, and risk of bias between published systematic reviews with reported AI assistance and matched reviews with no reported AI use, and to assess AI-use transparency. Methods: We conducted a matched-cohort meta-research study of English-language, peer-reviewed health systematic reviews. AI-assisted reviews published during 2023--2025 were matched 1:2 to reviews with no reported AI use using individualized MEDLINE searches targeting publication timing, clinical domain, review type, and meta-analysis status. When more than two eligible comparators were available, two were randomly selected. AI assistance was defined as reported use during searching, screening, data extraction, risk-of-bias assessment, evidence synthesis, or substantive drafting. Full texts and supplementary materials were examined to classify reported AI use. The primary outcome was a prespecified modified 13-item AMSTAR-2 methodological-adherence score (range 0 -- 13; noninferiority margin &-1 point). Secondary outcomes included PRISMA 2020 reporting completeness and ROBIS risk of bias; AI-use transparency was assessed descriptively. Adapted 13-item AMSTAR-2 confidence categories were summarized descriptively, while critical- and noncritical-domain adherence was examined in exploratory adjusted analyses. Multivariable regression models adjusted for journal impact tier, funding reporting, formal review-group affiliation, review-team size, and review-question type, using cluster-robust standard errors by matched set. Sensitivity analyses included matched-set fixed-effects models. The study was registered on the Open Science Framework (DOI: 10.17605/OSF.IO/Q5JRW). Results: The cohort included 231 systematic reviews: 77 AI-assisted and 154 with no reported AI use. Mean 13-item AMSTAR-2 scores were 7.58 and 9.63, respectively. In multivariable-adjusted analyses, AI-assisted reviews scored 1.64 points lower (95% CI &-2.37 to &- 0.91); noninferiority was not demonstrated. PRISMA adherence was estimated to be 4.7 percentage points lower among AI-assisted reviews in primary adjusted analyses (95% CI &-8.9 to &-0.6) and 6.4 points lower in matched-set fixed-effects analyses (95% CI &-10.7 to &-2.1). Among reviews with assessable ROBIS ratings, 75.0% of AI-assisted reviews and 65.6% of comparators were rated low risk (adjusted difference +17.8 percentage points; 95% CI 2.7 to 33.0). Among AI-assisted reviews, model version or date was reported in 42.9%, prompts or parameters in 55.8%, and data-governance or ethics considerations in 9.1%. Conclusions: Reviews reporting AI assistance had lower methodological-adherence scores and had less complete reporting, despite more favorable risk-of-bias ratings. These findings provide an early warning that AI-assisted systematic reviews may require stronger safeguards, human verification, and transparency to support trust.
ER -