{ "abstract": "Objectives: To evaluate the implementation and use of electronic health record (EHR)-integrated large language model (LLM)-generated draft responses in patient portal messaging, including adoption over time, draft reuse and associations with message handling time.\n\nDesign: One-year prospective observational study using in-system EHR log data, complemented by a post-implementation clinician survey.\n\nSetting: A tertiary academic hospital in the Netherlands.\n\nParticipants: Patient portal messages with clinician replies during the study period (March 2024–March 2025; n=10 786). A voluntary clinician survey was completed by 241 respondents.\n\nInterventions: An EHR-integrated LLM automatically generated draft replies for incoming patient portal messages. Clinicians could use, edit or discard drafts as part of routine clinical workflow.\n\nPrimary and secondary outcome measures: Primary outcomes were adoption of draft replies, textual similarity between drafts and final replies (Recall-Oriented Understudy for Gisting Evaluation (ROUGE-1) and Bilingual Evaluation Understudy (BLEU) scores), relative word-count change and message handling time metrics (read-to-action time, draft time and total completion time). Secondary outcomes included temporal trends in adoption, variation across clinical departments and clinician-reported experiences.\n\nResults: Draft replies were initiated in 42% of patient messages. Among drafted replies, 52% met the predefined substantial-retention threshold (BLEU ≥10). Messages with higher draft retention (BLEU ≥10) were associated with shorter median draft time (70 s (IQR 37–124)) and total handling time (140 s (IQR 81–232)) compared with messages with no draft initiation (86 s (IQR 47–154) and 155 s (IQR 87–258), respectively) or minimal draft retention (99 s (IQR 53–174) and 175 s (IQR 103–285)). Adoption declined over time to approximately 35–40% and varied across departments, with higher use in orthopaedics, otolaryngology and neurology and lower use in oncology, rheumatology and gastroenterology. Survey responses indicated mixed perceptions, with commonly reported benefits including a structured starting point and frequently cited limitations related to tone and contextual relevance.\n\nConclusions and relevance: EHR-integrated LLM-generated draft responses showed moderate and variable uptake and were reused to differing extents across messages and specialties. Meaningful draft reuse was associated with shorter message handling times, although adoption declined over time. These findings highlight the importance of workflow integration, specialty-specific fit and ongoing evaluation when implementing generative artificial intelligence tools in clinical communication.", "abstract_shown": true, "affiliations": [ "Department of Health Information Office, Information Management Healthcare, University Medical Centre Groningen, Groningen, The Netherlands", "Department of Pediatrics, Pediatric Nephrology, University Medical Centre Groningen, Groningen, The Netherlands", "Department of Trauma Surgery / Orthopedics, University Medical Centre Groningen, Groningen, The Netherlands", "Department of Orthopaedic Trauma, Flinders University, Adelaide, South Australia, Australia", "Department of Otolaryngology – Head and Neck Surgery, University Medical Centre Groningen, Groningen, The Netherlands", "Board of Directors, University Medical Centre Groningen, Groningen, The Netherlands" ], "article_type": "research article", "arxiv": null, "author_total": 7, "authors": [ { "affs": [ 0 ], "name": "Anne H Hoekman", "orcid": "0009-0006-5308-4211" }, { "affs": [ 0, 1 ], "name": "Charlotte MHHT Bootsma-Robroeks" }, { "affs": [ 0 ], "name": "Tarannom Mehri" }, { "affs": [ 2, 3 ], "name": "Job N Doornberg" }, { "affs": [ 4 ], "name": "Tom P van der Laan" }, { "affs": [ 5 ], "name": "Stephanie CE Schuit" }, { "affs": [ 0, 4 ], "name": "Rosanne C Schoonbeek", "orcid": "0000-0002-3103-943X" } ], "bibtex": "https://research.draconitic.com/paper/eq6vry0d73580.bib", "cited_by": 0, "doi": "10.1136/bmjopen-2026-119071", "fulltext_url": "https://pmc.ncbi.nlm.nih.gov/articles/PMC13723810/", "id": "eq6vry0d73580", "issns": [ "2044-6055", "2044-6055" ], "landing_url": "https://bmjopen.bmj.com/lookup/doi/10.1136/bmjopen-2026-119071", "language": "en", "licence": "CC-BY-NC-4.0", "licence_label": "CC BY-NC 4.0", "licence_source": "article", "licence_url": "https://creativecommons.org/licenses/by-nc/4.0/", "metadata_licence": null, "native_subjects": [ [ "heading", "1506" ], [ "heading", "1702" ], [ "heading", "Health Informatics" ], [ "heading", "Original Research" ] ], "openalex": null, "pdf_url": null, "pmcid": "PMC13723810", "pmid": "42963099", "published": "2026-10-30", "publisher": "BMJ", "ris": "https://research.draconitic.com/paper/eq6vry0d73580.ris", "sources": [ { "first_seen": "2026-10-31T00:21:03Z", "native_id": "10.1136/bmjopen-2026-119071", "source": "crossref", "url": "https://bmjopen.bmj.com/lookup/doi/10.1136/bmjopen-2026-119071" }, { "first_seen": "2026-11-04T12:22:22Z", "native_id": "PMC13723810", "source": "pmc", "url": "https://doi.org/10.1136/bmjopen-2026-119071" } ], "subjects": [ { "field": "medicine", "origin": "journal-openalex:F27", "subfield": "general" } ], "title": "Implementation and use of EHR-integrated large language model-generated draft responses in patient portal messaging: a 1-year prospective observational study in a tertiary hospital", "title_text": "Implementation and use of EHR-integrated large language model-generated draft responses in patient portal messaging: a 1-year prospective observational study in a tertiary hospital", "type": "peer-reviewed", "updated_at": "2026-11-05T12:20:33Z", "url": "https://research.draconitic.com/paper/eq6vry0d73580", "venue": "BMJ Open", "versions": [] }