Evaluating an LLM-assisted workflow for clinical documentation: A pilot randomized controlled trial on time and quality
7 affiliations
- 1Department of Ophthalmology and Visual Sciences, Kyoto University Graduate School of Medicine, Kyoto, Japan.
- 2Kyoto University Hospital, Kyoto, Japan.
- 3Fitting Cloud Inc., Kyoto, Japan.
- 4Center for Innovative Research and Education in Data Science, Institute for Liberal Arts and Sciences, Kyoto University, Kyoto, Japan.
- 5Japanese Society of Artificial Intelligence in Ophthalmology, Chuoku, Tokyo, Japan.
- 6Department of Ophthalmology, Faculty of Medicine, University of Tsukuba, Ibaraki, Japan.
- 7Division of Medical Information Technology and Administration Planning, Kyoto University Hospital, Kyoto University, Kyoto, Japan.
Abstract
Large language models (LLMs) have been investigated for clinical documentation, with concerns about hallucinations and factual errors. Clinician review and revision of LLM-generated drafts are considered essential; however, the impact of such a workflow on both documentation time and quality remains unknown. This single-center, parallel-group, prospective, randomized, open-label, blinded-endpoint (PROBE) pilot trial evaluated whether a clinician-in-the-loop workflow improves clinical documentation time and quality compared with clinician-only drafting. From February 18 to March 14, 2025, at the Department of Ophthalmology, Kyoto University Hospital, 21 physicians were randomized, and 17 completed the intervention. Participants were randomized to either the Clinician-in-the-loop group or the Clinician-only group. Participants created discharge summaries and referrals for six simulated patient records. The Clinician-in-the-loop group reviewed and edited LLM-generated drafts, whereas the Clinician-only group drafted documents from scratch using matched templates. Unedited LLM drafts formed the LLM-only group. Seventeen physicians submitted 48 discharge summaries and 48 discharge referrals in the Clinician-in-the-loop group, 54 of each document type in the Clinician-only group, and 48 of each in the LLM-only group. Clinician-in-the-loop showed a shorter creation time for summaries (β = −59.4 seconds; 95% CI, −150.6 to 31.7; P = .18) and a longer creation time for referrals (β = 52.9 seconds; 95% CI, −7.0 to 112.9; P = .08) than Clinician-only, although neither difference was statistically significant. In most expert-rated quality domains, the clinician-in-the-loop workflow outperformed clinician-only drafting. LLM-only drafts were fastest but had the lowest quality. In conclusion, a clinician-in-the-loop approach generally improved document quality and, although the primary analysis showed no statistically significant difference in documentation time, suggested potential time savings for certain document types. These findings highlight the importance of active clinician supervision and support further evaluation of clinician-in-the-loop workflows in real-world clinical settings. Trial Registration ClinicalTrials.gov Identifier NCT07187050 (registered on September 18, 2025). Author summary: Clinical documentation such as discharge summaries and discharge referrals is essential for patient care, yet it places a heavy burden on clinicians. Recently, large language models (LLMs) have been proposed as tools that might ease this burden, but it remains uncertain whether documents created with the help of an LLM can save time while still maintaining acceptable quality. In this study, we conducted a randomized controlled trial in which clinicians created clinical documents either by reviewing and editing drafts generated by an LLM or by writing them on their own. We also analyzed unedited LLM drafts to understand how the model performs without clinician review. We found that the clinician-in-the-loop approach generally improved document quality and may save time for some document types, although the primary analysis found no statistically significant difference in documentation time. In contrast, LLM-only drafts were produced most quickly but did not meet quality standards. These findings suggest that LLMs may support clinical documentation when paired with active clinician oversight. Such collaborative workflows warrant further evaluation in real-world clinical settings.
Related
- Evaluating the efficacy and safety of therapeutic plasma exchange in snakebite-associated thrombotic microangiopathy compared to standard care alone: study protocol for a multicenter randomized controlled trial
- Erector spinae plane block added to intercostal nerve block versus intercostal nerve block alone for postoperative analgesia in video-assisted thoracoscopic surgery: A randomized controlled trial
- A heat-assisted ttLbCas12a workflow incorporating crRNA 3'-tailing enables efficient multiplex biallelic genome editing in hybrid poplar
- Technology-Assisted Rehabilitation Gym for at-home Training (TARGeT) for stroke survivors: A study protocol for evaluating feasibility, safety, usability, clinical benefit, and costs
- Effects of myofascial release on clinical outcomes and muscle viscoelastic properties in patients with breast cancer: A randomized controlled trial
BibTeX
@article{takayama2026evaluating,
title = {{Evaluating an LLM-assisted workflow for clinical documentation: A pilot randomized controlled trial on time and quality}},
author = {Tomohiro Takayama and Keina Sado and Kenji Suda and Hiroshi Tamura and Naoko Ueda-Arakawa and Kenji Ishihara and Yuta Ueda and Kazuya Okamoto and Luciano Henrique de Oliveira Santos and Tetsuro Oshika and Tomohiro Kuroda and Akitaka Tsujikawa and Masahiro Miyake},
journal = {PLOS Digital Health},
year = {2026},
month = oct,
doi = {10.1371/journal.pdig.0001799},
url = {https://journals.plos.org/digitalhealth/article?id=10.1371/journal.pdig.0001799},
note = {CC BY},
}
RIS
TY - JOUR
TI - Evaluating an LLM-assisted workflow for clinical documentation: A pilot randomized controlled trial on time and quality
AU - Tomohiro Takayama
AU - Keina Sado
AU - Kenji Suda
AU - Hiroshi Tamura
AU - Naoko Ueda-Arakawa
AU - Kenji Ishihara
AU - Yuta Ueda
AU - Kazuya Okamoto
AU - Luciano Henrique de Oliveira Santos
AU - Tetsuro Oshika
AU - Tomohiro Kuroda
AU - Akitaka Tsujikawa
AU - Masahiro Miyake
PY - 2026
DA - 2026/10/26
JO - PLOS Digital Health
DO - 10.1371/journal.pdig.0001799
UR - https://journals.plos.org/digitalhealth/article?id=10.1371/journal.pdig.0001799
AB - Large language models (LLMs) have been investigated for clinical documentation, with concerns about hallucinations and factual errors. Clinician review and revision of LLM-generated drafts are considered essential; however, the impact of such a workflow on both documentation time and quality remains unknown. This single-center, parallel-group, prospective, randomized, open-label, blinded-endpoint (PROBE) pilot trial evaluated whether a clinician-in-the-loop workflow improves clinical documentation time and quality compared with clinician-only drafting. From February 18 to March 14, 2025, at the Department of Ophthalmology, Kyoto University Hospital, 21 physicians were randomized, and 17 completed the intervention. Participants were randomized to either the Clinician-in-the-loop group or the Clinician-only group. Participants created discharge summaries and referrals for six simulated patient records. The Clinician-in-the-loop group reviewed and edited LLM-generated drafts, whereas the Clinician-only group drafted documents from scratch using matched templates. Unedited LLM drafts formed the LLM-only group. Seventeen physicians submitted 48 discharge summaries and 48 discharge referrals in the Clinician-in-the-loop group, 54 of each document type in the Clinician-only group, and 48 of each in the LLM-only group. Clinician-in-the-loop showed a shorter creation time for summaries (β = −59.4 seconds; 95% CI, −150.6 to 31.7; P = .18) and a longer creation time for referrals (β = 52.9 seconds; 95% CI, −7.0 to 112.9; P = .08) than Clinician-only, although neither difference was statistically significant. In most expert-rated quality domains, the clinician-in-the-loop workflow outperformed clinician-only drafting. LLM-only drafts were fastest but had the lowest quality. In conclusion, a clinician-in-the-loop approach generally improved document quality and, although the primary analysis showed no statistically significant difference in documentation time, suggested potential time savings for certain document types. These findings highlight the importance of active clinician supervision and support further evaluation of clinician-in-the-loop workflows in real-world clinical settings. Trial Registration ClinicalTrials.gov Identifier NCT07187050 (registered on September 18, 2025). Author summary: Clinical documentation such as discharge summaries and discharge referrals is essential for patient care, yet it places a heavy burden on clinicians. Recently, large language models (LLMs) have been proposed as tools that might ease this burden, but it remains uncertain whether documents created with the help of an LLM can save time while still maintaining acceptable quality. In this study, we conducted a randomized controlled trial in which clinicians created clinical documents either by reviewing and editing drafts generated by an LLM or by writing them on their own. We also analyzed unedited LLM drafts to understand how the model performs without clinician review. We found that the clinician-in-the-loop approach generally improved document quality and may save time for some document types, although the primary analysis found no statistically significant difference in documentation time. In contrast, LLM-only drafts were produced most quickly but did not meet quality standards. These findings suggest that LLMs may support clinical documentation when paired with active clinician oversight. Such collaborative workflows warrant further evaluation in real-world clinical settings.
SN - 2767-3170
ER -