Skip to content
DraconiticResearch
Search Sources API
182,469 studies31 sourcesUpdated 15 min ago
Peer-reviewedCC BY-NC 4.0BMJ Open

Implementation and use of EHR-integrated large language model-generated draft responses in patient portal messaging: a 1-year prospective observational study in a tertiary hospital

  • Anne H Hoekman1
  • Charlotte MHHT Bootsma-Robroeks1,2
  • Tarannom Mehri1
  • Job N Doornberg3,4
  • Tom P van der Laan5
  • Stephanie CE Schuit6
  • Rosanne C Schoonbeek1,5
6 affiliations
  1. 1Department of Health Information Office, Information Management Healthcare, University Medical Centre Groningen, Groningen, The Netherlands
  2. 2Department of Pediatrics, Pediatric Nephrology, University Medical Centre Groningen, Groningen, The Netherlands
  3. 3Department of Trauma Surgery / Orthopedics, University Medical Centre Groningen, Groningen, The Netherlands
  4. 4Department of Orthopaedic Trauma, Flinders University, Adelaide, South Australia, Australia
  5. 5Department of Otolaryngology – Head and Neck Surgery, University Medical Centre Groningen, Groningen, The Netherlands
  6. 6Board of Directors, University Medical Centre Groningen, Groningen, The Netherlands
Read full textDOICite

Abstract

Objectives: To evaluate the implementation and use of electronic health record (EHR)-integrated large language model (LLM)-generated draft responses in patient portal messaging, including adoption over time, draft reuse and associations with message handling time.

Design: One-year prospective observational study using in-system EHR log data, complemented by a post-implementation clinician survey.

Setting: A tertiary academic hospital in the Netherlands.

Participants: Patient portal messages with clinician replies during the study period (March 2024–March 2025; n=10 786). A voluntary clinician survey was completed by 241 respondents.

Interventions: An EHR-integrated LLM automatically generated draft replies for incoming patient portal messages. Clinicians could use, edit or discard drafts as part of routine clinical workflow.

Primary and secondary outcome measures: Primary outcomes were adoption of draft replies, textual similarity between drafts and final replies (Recall-Oriented Understudy for Gisting Evaluation (ROUGE-1) and Bilingual Evaluation Understudy (BLEU) scores), relative word-count change and message handling time metrics (read-to-action time, draft time and total completion time). Secondary outcomes included temporal trends in adoption, variation across clinical departments and clinician-reported experiences.

Results: Draft replies were initiated in 42% of patient messages. Among drafted replies, 52% met the predefined substantial-retention threshold (BLEU ≥10). Messages with higher draft retention (BLEU ≥10) were associated with shorter median draft time (70 s (IQR 37–124)) and total handling time (140 s (IQR 81–232)) compared with messages with no draft initiation (86 s (IQR 47–154) and 155 s (IQR 87–258), respectively) or minimal draft retention (99 s (IQR 53–174) and 175 s (IQR 103–285)). Adoption declined over time to approximately 35–40% and varied across departments, with higher use in orthopaedics, otolaryngology and neurology and lower use in oncology, rheumatology and gastroenterology. Survey responses indicated mixed perceptions, with commonly reported benefits including a structured starting point and frequently cited limitations related to tone and contextual relevance.

Conclusions and relevance: EHR-integrated LLM-generated draft responses showed moderate and variable uptake and were reused to differing extents across messages and specialties. Meaningful draft reuse was associated with shorter message handling times, although adoption declined over time. These findings highlight the importance of workflow integration, specialty-specific fit and ongoing evaluation when implementing generative artificial intelligence tools in clinical communication.

Related

  1. Systemic, Pleural, and Alveolar Cytokine Responses After Video-Assisted Thoracoscopic Versus Open Lobectomy for Non-Small Cell Lung Cancer: A Prospective Observational Study
    Burcu Ancın, Mesut Melih Özercan, Yiğit Yılmaz +5
    Journal of clinical medicinePeer-reviewedCC BYFull textGeneral medicineEurope PMC
  2. Implementation of a structured early oral supportive-care pathway and early oral health recovery after oral cancer resection and free-flap reconstruction: a prospective observational cohort study
    Tsai-Mei Huang, Yun Chih Chen, Yu-Che Cheng +5
    Frontiers in OncologyPeer-reviewedCC BY 4.0Full textOncologyCrossrefEurope PMC
  3. Implementation of Clinical Pharmacy Activities in Pediatric Cardiology Department: A French Observational Study
    Gabrielle Robineau, Morgane Cabon, Thomas Perouse-De-Montclos +7
    Pediatric CardiologyPeer-reviewedLicence not statedFull textGeneral biologyEpidemiology & public healthHAL
  4. Navigating the transition to digital transfusion safety in a resource-limited setting: a prospective implementation study and operational workflow analysis.
    Ninnate Junta, Ronnarit Boonyarat, Chalawan Butryojantho +4
    BMJ open qualityPeer-reviewedCC BY-NCFull textGeneral medicineEurope PMC
  5. Virologic and Exploratory Immunologic Characterization of Relapsing Human Cytomegalovirus (HCMV) DNAemia in Solid Organ Transplant Recipients: A Prospective Observational Cohort Study
    Open forum infectious diseasesPeer-reviewedCC BYFull textGeneral medicineEurope PMC

Cite

BibTeXRIS
BibTeX
@article{hoekman2026implementation,
  title = {{Implementation and use of EHR-integrated large language model-generated draft responses in patient portal messaging: a 1-year prospective observational study in a tertiary hospital}},
  author = {Anne H Hoekman and Charlotte MHHT Bootsma-Robroeks and Tarannom Mehri and Job N Doornberg and Tom P van der Laan and Stephanie CE Schuit and Rosanne C Schoonbeek},
  journal = {BMJ Open},
  year = {2026},
  month = oct,
  doi = {10.1136/bmjopen-2026-119071},
  url = {https://bmjopen.bmj.com/lookup/doi/10.1136/bmjopen-2026-119071},
  note = {CC BY-NC 4.0},
}
RIS
TY  - JOUR
TI  - Implementation and use of EHR-integrated large language model-generated draft responses in patient portal messaging: a 1-year prospective observational study in a tertiary hospital
AU  - Anne H Hoekman
AU  - Charlotte MHHT Bootsma-Robroeks
AU  - Tarannom Mehri
AU  - Job N Doornberg
AU  - Tom P van der Laan
AU  - Stephanie CE Schuit
AU  - Rosanne C Schoonbeek
PY  - 2026
DA  - 2026/10/30
JO  - BMJ Open
DO  - 10.1136/bmjopen-2026-119071
UR  - https://bmjopen.bmj.com/lookup/doi/10.1136/bmjopen-2026-119071
AB  - Objectives: To evaluate the implementation and use of electronic health record (EHR)-integrated large language model (LLM)-generated draft responses in patient portal messaging, including adoption over time, draft reuse and associations with message handling time.  Design: One-year prospective observational study using in-system EHR log data, complemented by a post-implementation clinician survey.  Setting: A tertiary academic hospital in the Netherlands.  Participants: Patient portal messages with clinician replies during the study period (March 2024–March 2025; n=10 786). A voluntary clinician survey was completed by 241 respondents.  Interventions: An EHR-integrated LLM automatically generated draft replies for incoming patient portal messages. Clinicians could use, edit or discard drafts as part of routine clinical workflow.  Primary and secondary outcome measures: Primary outcomes were adoption of draft replies, textual similarity between drafts and final replies (Recall-Oriented Understudy for Gisting Evaluation (ROUGE-1) and Bilingual Evaluation Understudy (BLEU) scores), relative word-count change and message handling time metrics (read-to-action time, draft time and total completion time). Secondary outcomes included temporal trends in adoption, variation across clinical departments and clinician-reported experiences.  Results: Draft replies were initiated in 42% of patient messages. Among drafted replies, 52% met the predefined substantial-retention threshold (BLEU ≥10). Messages with higher draft retention (BLEU ≥10) were associated with shorter median draft time (70 s (IQR 37–124)) and total handling time (140 s (IQR 81–232)) compared with messages with no draft initiation (86 s (IQR 47–154) and 155 s (IQR 87–258), respectively) or minimal draft retention (99 s (IQR 53–174) and 175 s (IQR 103–285)). Adoption declined over time to approximately 35–40% and varied across departments, with higher use in orthopaedics, otolaryngology and neurology and lower use in oncology, rheumatology and gastroenterology. Survey responses indicated mixed perceptions, with commonly reported benefits including a structured starting point and frequently cited limitations related to tone and contextual relevance.  Conclusions and relevance: EHR-integrated LLM-generated draft responses showed moderate and variable uptake and were reused to differing extents across messages and specialties. Meaningful draft reuse was associated with shorter message handling times, although adoption declined over time. These findings highlight the importance of workflow integration, specialty-specific fit and ongoing evaluation when implementing generative artificial intelligence tools in clinical communication.
SN  - 2044-6055
SN  - 2044-6055
ER  -