Retrospective Comparison of Radiation Oncology Patient Message Responses Generated by Epic and Institutionally Managed LLMs
Abstract
We retrospectively compared Epic-generated draft replies with institutionally managed hybrid GPT-4.1 and Gemini 2.5 Pro responses to 51 selected single-turn radiation oncology messages. Six clinicians, blinded to response source, rated overall quality on a five-point scale and perceived editing effort on a four-point scale. Mixed-effects models accounted for repeated ratings by message and clinician. Adjusted overall-quality means were 2.95 for Epic, 4.27 for hybrid GPT-4.1, and 4.39 for hybrid Gemini 2.5 Pro. Corresponding editing-effort means were 2.38, 1.46, and 1.41, with lower scores indicating less perceived effort. Both hybrid models received more favorable ratings than Epic for both outcomes (all Holm-adjusted P < 0.001), without a statistically significant difference between the hybrid models. These findings support prospective evaluation of clinician-supervised draft generation, though editing time and patient outcomes were not measured, and clinical safety was not established.
Related
- Second-order difference-frequency effects on floating wind turbine responses: full-QTF modelling and experimental comparison
- Dosimetric Comparison of Intensity-Modulated Radiotherapy and Four-Field Three-Dimensional Conformal Radiotherapy in the Management of Prostate Cancer: A Retrospective Dual-Centre Analysis
- ExacTrac Dynamic Surface-Guided Setup Versus Tattoo-Based Positioning in Breast Cancer Radiotherapy: A Retrospective Comparison of Setup Accuracy and Preparation Time
- Photonically Generated Simultaneous Multi-Band Terahertz Communication System
- Analytical framework for computer-generated holography of screen space primitives with point-source integration in the Fourier domain
BibTeX
@misc{cao2026retrospective,
title = {{Retrospective Comparison of Radiation Oncology Patient Message Responses Generated by Epic and Institutionally Managed LLMs}},
author = {Meiyun Cao and Federico Mastroleo and Shuoyang Wei and Yuexing Hao and Srinivas Seetamsetty and Keldon K. Lin and Madhurya Reddy and Yuzhen Ding and Jason Holmes and Jonathan E. Moonen and Shyanne L. Dobbs and Diego Santos Toesca and Mark Waddle and Wei Liu},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.34464},
eprint = {2610.34464},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.34464},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Retrospective Comparison of Radiation Oncology Patient Message Responses Generated by Epic and Institutionally Managed LLMs
AU - Meiyun Cao
AU - Federico Mastroleo
AU - Shuoyang Wei
AU - Yuexing Hao
AU - Srinivas Seetamsetty
AU - Keldon K. Lin
AU - Madhurya Reddy
AU - Yuzhen Ding
AU - Jason Holmes
AU - Jonathan E. Moonen
AU - Shyanne L. Dobbs
AU - Diego Santos Toesca
AU - Mark Waddle
AU - Wei Liu
PY - 2026
DA - 2026/10/25
PB - arXiv
DO - 10.48550/arxiv.2610.34464
UR - https://arxiv.org/abs/2610.34464
AB - We retrospectively compared Epic-generated draft replies with institutionally managed hybrid GPT-4.1 and Gemini 2.5 Pro responses to 51 selected single-turn radiation oncology messages. Six clinicians, blinded to response source, rated overall quality on a five-point scale and perceived editing effort on a four-point scale. Mixed-effects models accounted for repeated ratings by message and clinician. Adjusted overall-quality means were 2.95 for Epic, 4.27 for hybrid GPT-4.1, and 4.39 for hybrid Gemini 2.5 Pro. Corresponding editing-effort means were 2.38, 1.46, and 1.41, with lower scores indicating less perceived effort. Both hybrid models received more favorable ratings than Epic for both outcomes (all Holm-adjusted P < 0.001), without a statistically significant difference between the hybrid models. These findings support prospective evaluation of clinician-supervised draft generation, though editing time and patient outcomes were not measured, and clinical safety was not established.
ER -