CHARMS and PROBAST+AI: an updated template for Data Extraction and Risk of Bias Assessment in systematic reviews of prediction models
- 1University of Glasgow
Abstract
Systematic reviews of clinical prediction models increasingly include studies using artificial intelligence (AI) and machine learning (ML) methods alongside traditional multivariable regression approaches. A previously published Excel tool enabled standardised data extraction using the CHARMS checklist and risk of bias assessment using PROBAST. The recent publication of the PROBAST+AI framework, which distinguishes the assessment of model development quality from the assessment of model evaluation risk of bias and assesses applicability in both parts, necessitates an updated digital instrument applicable across prediction modelling methods.
We updated an open-access Excel tool to incorporate the full PROBAST+AI framework. The updated template incorporates structural separation between assessment of model development quality and model evaluation risk of bias, with applicability assessed in both parts. It also incorporates updated signalling questions, including those addressing methodological issues particularly relevant to AI/ML, and automates the generation of summary tables and graphical displays.
The updated tool (CHARMS & PROBAST+AI Template) contains 11 worksheets and supports data extraction and appraisal for up to 30 prediction models. Dedicated, linked worksheets enable separate assessment of model development and model evaluation, with Domain 4 distinguishing among Apparent, Internal, and External evaluation settings. Key updates include dedicated assessments for predictor pre-processing, class imbalance handling and recalibration, data leakage prevention, and replication of the full model development pipeline within resampling procedures. Automated sheets dynamically format tables and summary charts covering PROBAST+AI parts.
The CHARMS & PROBAST+AI Excel template provides a standardised, user-friendly, and rigorous digital framework for systematic reviewers appraising traditional statistical and AI-driven clinical prediction models.
Related
- Evaluation of Elicit as an adjunct search, screening, and data extraction tool for systematic reviews
- Anorectal function after transanal natural orifice specimen extraction for colorectal cancer: a systematic review and meta-analysis.
- Risk assessment and prediction of myopia among school-age children in South China: protocol for a longitudinal follow-up study
- Application of Ultrasound-Assisted Extraction for Isolating Rhamnetin from Clove (Syzygium aromaticum) Flower Buds: Process Optimization, Identification, and Bioactivity Assessment
- Development and Population-Based Prognostic Utility Assessment of an ICD-10-CM-Based Prediction Model to Identify Left Ventricular Systolic Dysfunction: Discrimination Performance and Population-Based Evaluation in 83,357 Patients with Acute Coronary Syndrome
BibTeX
@misc{jaber2026charms,
title = {{CHARMS and PROBAST+AI: an updated template for Data Extraction and Risk of Bias Assessment in systematic reviews of prediction models}},
author = {A. Jaber and L. Hughes and A. C. Cameron and T. J. Quinn},
howpublished = {medRxiv},
year = {2026},
month = aug,
doi = {10.64898/2026.08.26.26361189},
url = {https://www.medrxiv.org/content/10.64898/2026.08.26.26361189v1},
note = {CC BY},
}
RIS
TY - UNPB
TI - CHARMS and PROBAST+AI: an updated template for Data Extraction and Risk of Bias Assessment in systematic reviews of prediction models
AU - A. Jaber
AU - L. Hughes
AU - A. C. Cameron
AU - T. J. Quinn
PY - 2026
DA - 2026/08/26
PB - medRxiv
DO - 10.64898/2026.08.26.26361189
UR - https://www.medrxiv.org/content/10.64898/2026.08.26.26361189v1
AB - Systematic reviews of clinical prediction models increasingly include studies using artificial intelligence (AI) and machine learning (ML) methods alongside traditional multivariable regression approaches. A previously published Excel tool enabled standardised data extraction using the CHARMS checklist and risk of bias assessment using PROBAST. The recent publication of the PROBAST+AI framework, which distinguishes the assessment of model development quality from the assessment of model evaluation risk of bias and assesses applicability in both parts, necessitates an updated digital instrument applicable across prediction modelling methods. We updated an open-access Excel tool to incorporate the full PROBAST+AI framework. The updated template incorporates structural separation between assessment of model development quality and model evaluation risk of bias, with applicability assessed in both parts. It also incorporates updated signalling questions, including those addressing methodological issues particularly relevant to AI/ML, and automates the generation of summary tables and graphical displays. The updated tool (CHARMS & PROBAST+AI Template) contains 11 worksheets and supports data extraction and appraisal for up to 30 prediction models. Dedicated, linked worksheets enable separate assessment of model development and model evaluation, with Domain 4 distinguishing among Apparent, Internal, and External evaluation settings. Key updates include dedicated assessments for predictor pre-processing, class imbalance handling and recalibration, data leakage prevention, and replication of the full model development pipeline within resampling procedures. Automated sheets dynamically format tables and summary charts covering PROBAST+AI parts. The CHARMS & PROBAST+AI Excel template provides a standardised, user-friendly, and rigorous digital framework for systematic reviewers appraising traditional statistical and AI-driven clinical prediction models.
ER -