Prospective evaluation of a real-time AI assistant for outpatient documentation and quality control
Abstract
Outpatient electronic medical record (EMR) quality is highly variable, yet few tools exist to support real-time documentation. Here we developed SMART-ASSISTANT, an end-to-end AI system that automates EMR writing for outpatients through integrated audio transcription, structured EMR generation, quality control, and assisted diagnosis. Using abdominal pain as the exemplar, model performance was validated in simulated, retrospective, multi-reader multi-case (MRMC), and prospective cohort studies. SMART-ASSISTANT outperformed GPT-4 in identifying key clinical features (symptoms: 100.00% vs 19.20%; onset characteristics: 85.60% vs 45.60%; disease progression: 89.60% vs 45.40%; all P <0.001). In the MRMC study, physicians rated AI-assisted EMRs significantly higher than human-generated ones for completeness (4.27 vs 3.92, P <0.001) and diagnostic correlation (2.53 vs 2.33, P <0.001). In the prospective cohort, AI-generated EMRs showed higher integrity than physician-generated EMRs (3.50 vs 3.21, P =0.008), but lower factuality (3.34 vs 4.01, P <0.001), with no significant differences in normativity, readability, or logicality. SMART-ASSISTANT may support outpatient documentation by improving information completeness, but physician verification remains essential. Trial registration : Chinese Clinical Trial Registry, ChiCTR2400086606. Registered 08 July 2024, https://www.chictr.org.cn/ .
Related
- Effect of In-line Clinical Decision Support to Improve Accuracy of American Society of Anesthesiologists Physical Status Documentation: A Two Center Evaluation of a Quality Intervention
- Improving Urology Ward Round Documentation Through Sequential Digital Interventions: A Three-Cycle Quality Improvement Project
- Failed quality improvement project to improve adherence to CAM-ICU and RASS documentation in a tertiary intensive care unit: a cautionary tale.
- The Recovery Room Is Not Needed Following Hand Surgery With Local Anesthesia in the Outpatient Operating Room: A Prospective Study
- Changes in Psychological Burden and Quality of Life During In Vivo Diagnostic Evaluation for Immediate Drug Hypersensitivity: A Prospective Stepwise Study
BibTeX
@article{du2026prospective,
title = {{Prospective evaluation of a real-time AI assistant for outpatient documentation and quality control}},
author = {Hongliu Du and Jialing Li and Bing Xiao and Xueying Wang and Wenxin Xue and Mei Deng and Shuzhe Tan and Jiamin Wang and Chaijie Luo and Boru Chen and Ting Yang and Zhan Chen and Qinxuan Zu and Joseph J. Y. Sung and Honggang Yu},
journal = {Scientific Reports},
year = {2026},
month = oct,
doi = {10.1038/s41598-026-74833-5},
url = {https://www.nature.com/articles/s41598-026-74833-5},
note = {CC BY 4.0},
}
RIS
TY - JOUR
TI - Prospective evaluation of a real-time AI assistant for outpatient documentation and quality control
AU - Hongliu Du
AU - Jialing Li
AU - Bing Xiao
AU - Xueying Wang
AU - Wenxin Xue
AU - Mei Deng
AU - Shuzhe Tan
AU - Jiamin Wang
AU - Chaijie Luo
AU - Boru Chen
AU - Ting Yang
AU - Zhan Chen
AU - Qinxuan Zu
AU - Joseph J. Y. Sung
AU - Honggang Yu
PY - 2026
DA - 2026/10/09
JO - Scientific Reports
DO - 10.1038/s41598-026-74833-5
UR - https://www.nature.com/articles/s41598-026-74833-5
AB - Outpatient electronic medical record (EMR) quality is highly variable, yet few tools exist to support real-time documentation. Here we developed SMART-ASSISTANT, an end-to-end AI system that automates EMR writing for outpatients through integrated audio transcription, structured EMR generation, quality control, and assisted diagnosis. Using abdominal pain as the exemplar, model performance was validated in simulated, retrospective, multi-reader multi-case (MRMC), and prospective cohort studies. SMART-ASSISTANT outperformed GPT-4 in identifying key clinical features (symptoms: 100.00% vs 19.20%; onset characteristics: 85.60% vs 45.60%; disease progression: 89.60% vs 45.40%; all P <0.001). In the MRMC study, physicians rated AI-assisted EMRs significantly higher than human-generated ones for completeness (4.27 vs 3.92, P <0.001) and diagnostic correlation (2.53 vs 2.33, P <0.001). In the prospective cohort, AI-generated EMRs showed higher integrity than physician-generated EMRs (3.50 vs 3.21, P =0.008), but lower factuality (3.34 vs 4.01, P <0.001), with no significant differences in normativity, readability, or logicality. SMART-ASSISTANT may support outpatient documentation by improving information completeness, but physician verification remains essential. Trial registration : Chinese Clinical Trial Registry, ChiCTR2400086606. Registered 08 July 2024, https://www.chictr.org.cn/ .
SN - 2045-2322
ER -