A Framework to Monitor Editing of Artificial Intelligence-Generated Medical Documentation
- 1University of California San Francisco
Abstract
ObjectiveTo develop and evaluate a framework for characterizing clinician editing of Artificial Intelligence (AI)-generated documentation and assess its feasibility for health system-level monitoring of AI scribes.
Materials and MethodsWe analyzed outpatient encounters in which an AI scribe was used at a single academic health system, examining the History of Present Illness (HPI) and Assessment and Plan (A&P) sections of notes. We characterized edits on three dimensions: lexical edit intensity based on Levenshtein distance, embedding edit intensity BERTScore, and clinical edit intensity based on removed and added UMLS concepts. We analyzed the Positive Predictive Value (PPV) of clinical edit intensity as a measure of clinically meaningful editing using clinicians as the gold standard, examined correlations among dimensions, and designed exponentially weighted moving-average control charts to monitor longitudinal changes in clinician editing behavior.
Results268,379 encounters were included (267,654 HPI, 267,594 A&P). Clinical edit intensity [≥]1 had an 88.9% PPV for clinically meaningful editing. Lexical and embedding edit intensity were highly correlated (Spearman {rho} 0.95), while clinical edit intensity was less strongly correlated with both ({rho} 0.77-0.81). Longitudinal monitoring detected changes coinciding with system-wide rollout.
DiscussionClinicians edited A&Ps more heavily than HPIs, potentially reflecting greater attention to content involving clinical decision-making. Over one-third of sections involved clinical concept changes, and clinical edit intensity identified clinically meaningful edits while providing information complementary to lexical editing measures.
ConclusionClinician editing can be characterized at scale using complementary editing dimensions, providing a scalable signal for post-deployment surveillance of the human-AI documentation process.
Related
- From plausible menus to trustworthy nutrition: a risk-proportionate VALID-Diet framework for artificial intelligence-generated dietary advice
- A prebill control plane for record integrity in inpatient artificial intelligence-enabled clinical documentation workflows.
- The Influence of Large Language Model-Generated Artificial Intelligence Risk Estimates on Clinical Decision-Making in Breast Imaging: A Simulation-Based Study
- Artificial Intelligence-generated Clinical Images for Plastic Surgery Education: A Prospective Randomized Double-blind Study.
- Artificial intelligence-generated and algorithmically personalized music interventions and their effects on brain function, behavior, and health: a critical review
BibTeX
@misc{garciaagundez2026framework,
title = {{A Framework to Monitor Editing of Artificial Intelligence-Generated Medical Documentation}},
author = {A. Garcia-Agundez and S. Zhou and J. Pourian and C. Blebea and P. Daneshpajouhnejad and E. Dente and K. Shi and S. Pollet and F. Xia and X. Shi and R. Thombley and C. Fenton and S. G. Murray and J. Adler-Milstein and G. Schmajuk and J. Feng and J. Yazdany},
howpublished = {medRxiv},
year = {2026},
month = sep,
doi = {10.64898/2026.09.30.26364427},
url = {https://www.medrxiv.org/content/10.64898/2026.09.30.26364427v1},
note = {CC BY-NC-ND},
}
RIS
TY - UNPB
TI - A Framework to Monitor Editing of Artificial Intelligence-Generated Medical Documentation
AU - A. Garcia-Agundez
AU - S. Zhou
AU - J. Pourian
AU - C. Blebea
AU - P. Daneshpajouhnejad
AU - E. Dente
AU - K. Shi
AU - S. Pollet
AU - F. Xia
AU - X. Shi
AU - R. Thombley
AU - C. Fenton
AU - S. G. Murray
AU - J. Adler-Milstein
AU - G. Schmajuk
AU - J. Feng
AU - J. Yazdany
PY - 2026
DA - 2026/09/30
PB - medRxiv
DO - 10.64898/2026.09.30.26364427
UR - https://www.medrxiv.org/content/10.64898/2026.09.30.26364427v1
AB - ObjectiveTo develop and evaluate a framework for characterizing clinician editing of Artificial Intelligence (AI)-generated documentation and assess its feasibility for health system-level monitoring of AI scribes. Materials and MethodsWe analyzed outpatient encounters in which an AI scribe was used at a single academic health system, examining the History of Present Illness (HPI) and Assessment and Plan (A&P) sections of notes. We characterized edits on three dimensions: lexical edit intensity based on Levenshtein distance, embedding edit intensity BERTScore, and clinical edit intensity based on removed and added UMLS concepts. We analyzed the Positive Predictive Value (PPV) of clinical edit intensity as a measure of clinically meaningful editing using clinicians as the gold standard, examined correlations among dimensions, and designed exponentially weighted moving-average control charts to monitor longitudinal changes in clinician editing behavior. Results268,379 encounters were included (267,654 HPI, 267,594 A&P). Clinical edit intensity [≥]1 had an 88.9% PPV for clinically meaningful editing. Lexical and embedding edit intensity were highly correlated (Spearman {rho} 0.95), while clinical edit intensity was less strongly correlated with both ({rho} 0.77-0.81). Longitudinal monitoring detected changes coinciding with system-wide rollout. DiscussionClinicians edited A&Ps more heavily than HPIs, potentially reflecting greater attention to content involving clinical decision-making. Over one-third of sections involved clinical concept changes, and clinical edit intensity identified clinically meaningful edits while providing information complementary to lexical editing measures. ConclusionClinician editing can be characterized at scale using complementary editing dimensions, providing a scalable signal for post-deployment surveillance of the human-AI documentation process.
ER -