Quality and transcript-grounded clinical-risk error burden of AI-generated vs allied health professional clinical notes: a paired simulation study across four disciplines
- 1Heidi Health
Abstract
BackgroundAmbient artificial-intelligence (AI) scribes are being deployed beyond medicine into allied-health practice, but comparative evidence mainly concerns physician documentation.
ObjectiveTo compare quality and transcript-grounded clinical-risk error burden of AI-generated and allied health professional notes from identical simulated consultations.
MethodsPaired simulation across dietetics, physiotherapy, occupational therapy and speech and language therapy (five consultations each; 20 pairs; 40 notes). Heidi generated an unedited first-draft note and a professional documented each encounter. Professionals rated PDQI-9 quality. Professional and automated review identified unsupported statements and omissions; two language models graded clinical risk. Co-primary outcomes were PDQI-9 score and Critical+High error density.
ResultsMean PDQI-9 scores were 38.45 and 35.05 for AI and professional notes (paired difference +3.40, 95% CI +0.01 to +6.79; Wilcoxon p=0.042; dz=0.47). The Bayesian estimate was +3.47 (94% equal-tailed credible interval +0.66 to +6.30; P[AI>professional]=0.989). Automated-reviewer all-error density was 3.35 vs 8.55 per note (professional-minus-AI difference 5.20, 95% CI 3.85 to 6.55); omission density was 1.45 vs 5.20 (difference 3.75, 2.80 to 4.65). Critical+High density was 0.35 vs 0.75 (difference 0.40, -0.10 to 1.00). The pooled quality result was driven principally by dietetics.
ConclusionsAI-generated notes achieved modestly higher pooled quality and fewer transcript-grounded errors, chiefly omissions; serious-error evidence was inconclusive. Small discipline samples and incomplete ground truth preclude safety or discipline-wide superiority claims.
Related
- The Influence of Large Language Model-Generated Artificial Intelligence Risk Estimates on Clinical Decision-Making in Breast Imaging: A Simulation-Based Study
- Nutritional Accuracy and Clinical Applicability of Large Language Model-Generated Therapeutic Diet Plans for Phenylketonuria: A Comparative Simulation Study
- Auditing Large Language Model-Generated Digital Standardized Patients for Demographic Bias: A Simulation Study with HIV Pre-Exposure Prophylaxis Screening as a Tracer Condition
- Vaccine hesitancy across educational stages and health disciplines among healthcare students at a public university in the Western Black Sea region of Türkiye: a cross-sectional study
- Longitudinal Mechanism-of-Disease Mapping to Promote Integration Across Disciplines in Preclinical Medical Education.
BibTeX
@misc{bergman2026quality,
title = {{Quality and transcript-grounded clinical-risk error burden of AI-generated vs allied health professional clinical notes: a paired simulation study across four disciplines}},
author = {H. I. Bergman and V. Liu and S. Ali},
howpublished = {medRxiv},
year = {2026},
month = oct,
doi = {10.64898/2026.10.16.26365719},
url = {https://www.medrxiv.org/content/10.64898/2026.10.16.26365719v1},
note = {CC BY-NC-ND},
}
RIS
TY - UNPB
TI - Quality and transcript-grounded clinical-risk error burden of AI-generated vs allied health professional clinical notes: a paired simulation study across four disciplines
AU - H. I. Bergman
AU - V. Liu
AU - S. Ali
PY - 2026
DA - 2026/10/16
PB - medRxiv
DO - 10.64898/2026.10.16.26365719
UR - https://www.medrxiv.org/content/10.64898/2026.10.16.26365719v1
AB - BackgroundAmbient artificial-intelligence (AI) scribes are being deployed beyond medicine into allied-health practice, but comparative evidence mainly concerns physician documentation. ObjectiveTo compare quality and transcript-grounded clinical-risk error burden of AI-generated and allied health professional notes from identical simulated consultations. MethodsPaired simulation across dietetics, physiotherapy, occupational therapy and speech and language therapy (five consultations each; 20 pairs; 40 notes). Heidi generated an unedited first-draft note and a professional documented each encounter. Professionals rated PDQI-9 quality. Professional and automated review identified unsupported statements and omissions; two language models graded clinical risk. Co-primary outcomes were PDQI-9 score and Critical+High error density. ResultsMean PDQI-9 scores were 38.45 and 35.05 for AI and professional notes (paired difference +3.40, 95% CI +0.01 to +6.79; Wilcoxon p=0.042; dz=0.47). The Bayesian estimate was +3.47 (94% equal-tailed credible interval +0.66 to +6.30; P[AI>professional]=0.989). Automated-reviewer all-error density was 3.35 vs 8.55 per note (professional-minus-AI difference 5.20, 95% CI 3.85 to 6.55); omission density was 1.45 vs 5.20 (difference 3.75, 2.80 to 4.65). Critical+High density was 0.35 vs 0.75 (difference 0.40, -0.10 to 1.00). The pooled quality result was driven principally by dietetics. ConclusionsAI-generated notes achieved modestly higher pooled quality and fewer transcript-grounded errors, chiefly omissions; serious-error evidence was inconclusive. Small discipline samples and incomplete ground truth preclude safety or discipline-wide superiority claims.
ER -