Reporting Gaps and Challenges in Randomized Trials of Generative AI for Medical Use: Insights From a Systematic Review.
- 1Department of Regulatory Science for Medical Deivce, Dongguk University, Pildong-ro 1gil 30, Jung-gu, Seoul, 04620, Republic of Korea, 82 319615616.
- 2Department of Medical Device and Healthcare, Dongguk University, Seoul, Republic of Korea.
- 3Department of Medical AI Engineering, College of Advanced Convergence Engineering, Dongguk University, Seoul, Republic of Korea.
Abstract
Background
Generative AI (GenAI) is being deployed rapidly in medicine, and evidence of its clinical safety and efficacy depends on transparent reporting of randomized controlled trials (RCTs). The CONSORT-AI (Consolidated Standards of Reporting Trials-AI) extension is a benchmark for AI trial reporting, but it was drafted for conventional AI, and its applicability to GenAI has not been examined.Objective
This study aimed to assess item-level adherence to the CONSORT-AI among RCTs evaluating GenAI for medical purposes and to distinguish deficiencies attributable to nonadherence from those attributable to items that do not correspond to how GenAI interventions are specified or delivered.Methods
PubMed, CENTRAL (Cochrane Central Register of Controlled Trials), and Embase were searched for peer-reviewed, English-language RCTs published from 2022 to April 3, 2026, supplemented by backward and forward citation searching, in which patients, health care professionals, or laypeople received a GenAI intervention for a medical purpose; non-GenAI, nonmedical, nonrandomized, and non-peer-reviewed reports were excluded. Two reviewers independently screened studies, extracted data, and rated CONSORT-AI items as reported, partially reported, not reported, or not applicable using signaling questions, with disagreements resolved by a third reviewer (Cohen κ=0.81 and prevalence-adjusted, bias-adjusted κ=0.84). Risk of bias was not assessed, as the target was reporting adherence. Adherence was the proportion of reports rated "reported" per item; item-level adherence was compared across 9 characteristics using Fisher exact and Freeman-Halton tests (2-sided α=.05).Results
From 16,485 retrieved records, 74 reports from 73 RCTs (24,150 participants; median 100, IQR 50.0-247.0) were included. Large language models (LLMs) accounted for 98.6% (73/74) of interventions, with 60.8% (45/74) using GPT-family models. Item-level reporting adherence ranged from 0% (0/72) to 98.6% (73/74). High adherence occurred in general metadata and output specification items (range 59-73, 79.7%-98.6%). Conversely, reporting was poor for input and technical execution domains, input data eligibility (3/73, 4.1%), input acquisition/selection (9/74, 12.2%), algorithm versioning (17/74, 23.0%), trial setting integration (21/74, 28.4%), human-AI interaction (25/74, 33.8%), performance error (27/74, 36.5%), and open access/code (31/74, 41.9%). Input quality handling (Item 5[iii]) was completely unreported (0/72). Only 9 of 74 (12.2%) reports explicitly cited CONSORT-AI. While citing reports showed higher compliance for interaction and data sharing, major technical gaps persisted regardless of citation.Conclusions
This review provides the first empirical audit distinguishing general nonadherence from structural mismatches where deterministic CONSORT-AI items fail to capture GenAI mechanics. Differing from traditional AI evaluations, it reveals that key reproducibility drivers-system prompts, inference settings, and access modalities-are omitted from current standards. These empirical findings highlight critical reporting gaps in medical GenAI trials and identify domains for a tailored extension to current reporting standards, although the evidence is limited by the use of nonvalidated signaling questions. Establishing GenAI-specific trial guidelines will ensure that clinical decisions, regulatory approvals, and patient safety safeguards are grounded in reproducible and trustworthy evidence.
Related
- HER2-low and HER2-ultralow breast cancer in the antibody-drug conjugate era: diagnostic challenges and reporting workflow.
- Exercise for tension-type headache: protocol for a systematic review and meta-analysis with meta-regression and assessment of exercise reporting completeness.
- Gender-sensitive health reporting in the EU and OECD: insights from a cross-country review
- Completion, publication, and outcome reporting of registered randomized trials in abdominal surgery: registry-based cohort study.
- Evaluating a biochemical TB medication adherence test: Insights from image-based reporting through a patient companion app
BibTeX
@article{ko2026reporting,
title = {{Reporting Gaps and Challenges in Randomized Trials of Generative AI for Medical Use: Insights From a Systematic Review.}},
author = {Jae Eun Ko and Ji Woo Hwang and Sung Min Kim and Ji Yean Kwon},
journal = {Journal of medical Internet research},
year = {2026},
month = oct,
doi = {10.2196/94543},
url = {https://europepmc.org/article/MED/42955350},
note = {CC BY},
}
RIS
TY - JOUR
TI - Reporting Gaps and Challenges in Randomized Trials of Generative AI for Medical Use: Insights From a Systematic Review.
AU - Jae Eun Ko
AU - Ji Woo Hwang
AU - Sung Min Kim
AU - Ji Yean Kwon
PY - 2026
DA - 2026/10/01
JO - Journal of medical Internet research
DO - 10.2196/94543
UR - https://europepmc.org/article/MED/42955350
AB - Background Generative AI (GenAI) is being deployed rapidly in medicine, and evidence of its clinical safety and efficacy depends on transparent reporting of randomized controlled trials (RCTs). The CONSORT-AI (Consolidated Standards of Reporting Trials-AI) extension is a benchmark for AI trial reporting, but it was drafted for conventional AI, and its applicability to GenAI has not been examined.Objective This study aimed to assess item-level adherence to the CONSORT-AI among RCTs evaluating GenAI for medical purposes and to distinguish deficiencies attributable to nonadherence from those attributable to items that do not correspond to how GenAI interventions are specified or delivered.Methods PubMed, CENTRAL (Cochrane Central Register of Controlled Trials), and Embase were searched for peer-reviewed, English-language RCTs published from 2022 to April 3, 2026, supplemented by backward and forward citation searching, in which patients, health care professionals, or laypeople received a GenAI intervention for a medical purpose; non-GenAI, nonmedical, nonrandomized, and non-peer-reviewed reports were excluded. Two reviewers independently screened studies, extracted data, and rated CONSORT-AI items as reported, partially reported, not reported, or not applicable using signaling questions, with disagreements resolved by a third reviewer (Cohen κ=0.81 and prevalence-adjusted, bias-adjusted κ=0.84). Risk of bias was not assessed, as the target was reporting adherence. Adherence was the proportion of reports rated "reported" per item; item-level adherence was compared across 9 characteristics using Fisher exact and Freeman-Halton tests (2-sided α=.05).Results From 16,485 retrieved records, 74 reports from 73 RCTs (24,150 participants; median 100, IQR 50.0-247.0) were included. Large language models (LLMs) accounted for 98.6% (73/74) of interventions, with 60.8% (45/74) using GPT-family models. Item-level reporting adherence ranged from 0% (0/72) to 98.6% (73/74). High adherence occurred in general metadata and output specification items (range 59-73, 79.7%-98.6%). Conversely, reporting was poor for input and technical execution domains, input data eligibility (3/73, 4.1%), input acquisition/selection (9/74, 12.2%), algorithm versioning (17/74, 23.0%), trial setting integration (21/74, 28.4%), human-AI interaction (25/74, 33.8%), performance error (27/74, 36.5%), and open access/code (31/74, 41.9%). Input quality handling (Item 5[iii]) was completely unreported (0/72). Only 9 of 74 (12.2%) reports explicitly cited CONSORT-AI. While citing reports showed higher compliance for interaction and data sharing, major technical gaps persisted regardless of citation.Conclusions This review provides the first empirical audit distinguishing general nonadherence from structural mismatches where deterministic CONSORT-AI items fail to capture GenAI mechanics. Differing from traditional AI evaluations, it reveals that key reproducibility drivers-system prompts, inference settings, and access modalities-are omitted from current standards. These empirical findings highlight critical reporting gaps in medical GenAI trials and identify domains for a tailored extension to current reporting standards, although the evidence is limited by the use of nonvalidated signaling questions. Establishing GenAI-specific trial guidelines will ensure that clinical decisions, regulatory approvals, and patient safety safeguards are grounded in reproducible and trustworthy evidence.
SN - 1439-4456
SN - 1438-8871
ER -