A medically grounded LLM agent–based tool to detect patient safety events in medical records
8 affiliations
- 1Center for Learning Health Care Delivery, Beth Israel Deaconess Medical Center, Boston, Massachusetts, United States of America
- 2Department of Obstetrics Gynecology and Reproductive Medicine, Beth Israel Deaconess Medical Center and Harvard Medical School, Boston, Massachusetts, United States of America
- 3McWilliams School of Biomedical Informatics, UTHealth, Houston, Texas, United States of America
- 4Department of Emergency Medicine, Oregon Health and Science University, Portland, Oregon, United States of America
- 5Department of Biomedical Informatics, Harvard Medical School, Boston, Massachusetts, United States of America
- 6Department of Pediatrics and Emergency Medicine, University of British Columbia, Vancouver, Canada
- 7Department of Pediatrics, Oregon Health and Science University, Portland, Oregon, United States of America
- 8Department of Medical Informatics and Clinical Epidemiology, Oregon Health and Science University, Portland, Oregon, United States of America
Abstract
Large language models (LLMs) have shown incredible promise in medicine. While LLMs may be particularly useful in areas requiring extensive review of clinical records, their use remains limited due to their tendency to hallucinate and fabricate information. Hallucination issues, as well as their consequences, are exacerbated in low–probability, high–stakes scenarios such as rare adverse safety events or medical errors. We present SAFE–AI (Structured and Automated Framework for Explainable AI), a novel method for clinical decision making that combines the strengths of clinical expert knowledge with LLMs in an ontology–driven model that minimizes hallucinations using strict rules. We test this method to identify medication errors in medical charts. We collected a sample of 18,402 lines of clinical information from 300 EMS clinical charts that were independently dually reviewed by two expert physicians for epinephrine adverse safety events (ASEs), with 96% inter-rater agreement. We tested SAFE–AI against these labels, achieving similar performance to human experts in detecting epinephrine overdoses with 97.9% accuracy, and 91.6% accuracy in identifying delays in epinephrine administration, greatly outperforming baseline LLMs models. Notably, some disagreements between clinicians and the model were found to be justifiable differences in judgment rather than errors. SAFE-AI presents a novel approach for clinical AI applications that addresses two key limitations of current machine learning methods: 1) over-reliance on probabilistic pattern recognition instead of established medical knowledge, and 2) perpetuation of biases present in training data. This framework is easily adaptable to a range of clinical applications, paving the way for provable and trustworthy AI in medicine. Author summary: LLMs have shown promise in analyzing clinical records but their use is limited due to their tendency to hallucinate and fabricate information. Misinformation could threaten patient safety and jeopardize trust. We developed SAFE–AI (Structured and Automated Framework for Explainable AI), which combines knowledge from clinical experts with LLM inference to detect adverse safety events (ASEs) with minimal errors. We tested our method in identifying medication errors in medical charts and compared results to reviews by expert physicians. Our method detected epinephrine delays and overdoses with a high level of accuracy. SAFE-AI presents a novel approach for clinical AI applications that overcomes reliance on pattern recognition instead of medical knowledge biases present in training data.
Related
- Agentic LLM reasoning in pediatric diabetes outcome prediction from multimodal medical records
- Patient Trajectories from Electronic Health Records Suggest Specialty Specific Warning Signs for Common Variable Immunodeficiency
- Implementation of a Health History Disease Risk Instrument in Clinics Serving Medically Marginalized Patients
- Prevalent knowledge omissions: An unseen risk of LLMs in patient-friendly adaptation of medical literature
- Multispectral image analysis approach to detect adulteration of beef and pork in raw meats
BibTeX
@article{trujillo2026medically,
title = {{A medically grounded LLM agent–based tool to detect patient safety events in medical records}},
author = {Diego Trujillo and Dulin Wang and Nathan Bahr and Tina Yi-Jin Hsieh and Byeongyeon Cho and Garth Meckler and Matthew Hansen and Carl Eriksson and Kyu Seo Kim and Steven Bedrick and Xiaoqian Jiang and Jeanne-Marie Guise},
journal = {PLOS Digital Health},
year = {2026},
month = sep,
doi = {10.1371/journal.pdig.0001174},
url = {https://journals.plos.org/digitalhealth/article?id=10.1371/journal.pdig.0001174},
note = {CC BY 4.0},
}
RIS
TY - JOUR
TI - A medically grounded LLM agent–based tool to detect patient safety events in medical records
AU - Diego Trujillo
AU - Dulin Wang
AU - Nathan Bahr
AU - Tina Yi-Jin Hsieh
AU - Byeongyeon Cho
AU - Garth Meckler
AU - Matthew Hansen
AU - Carl Eriksson
AU - Kyu Seo Kim
AU - Steven Bedrick
AU - Xiaoqian Jiang
AU - Jeanne-Marie Guise
PY - 2026
DA - 2026/09/17
JO - PLOS Digital Health
DO - 10.1371/journal.pdig.0001174
UR - https://journals.plos.org/digitalhealth/article?id=10.1371/journal.pdig.0001174
AB - Large language models (LLMs) have shown incredible promise in medicine. While LLMs may be particularly useful in areas requiring extensive review of clinical records, their use remains limited due to their tendency to hallucinate and fabricate information. Hallucination issues, as well as their consequences, are exacerbated in low–probability, high–stakes scenarios such as rare adverse safety events or medical errors. We present SAFE–AI (Structured and Automated Framework for Explainable AI), a novel method for clinical decision making that combines the strengths of clinical expert knowledge with LLMs in an ontology–driven model that minimizes hallucinations using strict rules. We test this method to identify medication errors in medical charts. We collected a sample of 18,402 lines of clinical information from 300 EMS clinical charts that were independently dually reviewed by two expert physicians for epinephrine adverse safety events (ASEs), with 96% inter-rater agreement. We tested SAFE–AI against these labels, achieving similar performance to human experts in detecting epinephrine overdoses with 97.9% accuracy, and 91.6% accuracy in identifying delays in epinephrine administration, greatly outperforming baseline LLMs models. Notably, some disagreements between clinicians and the model were found to be justifiable differences in judgment rather than errors. SAFE-AI presents a novel approach for clinical AI applications that addresses two key limitations of current machine learning methods: 1) over-reliance on probabilistic pattern recognition instead of established medical knowledge, and 2) perpetuation of biases present in training data. This framework is easily adaptable to a range of clinical applications, paving the way for provable and trustworthy AI in medicine. Author summary: LLMs have shown promise in analyzing clinical records but their use is limited due to their tendency to hallucinate and fabricate information. Misinformation could threaten patient safety and jeopardize trust. We developed SAFE–AI (Structured and Automated Framework for Explainable AI), which combines knowledge from clinical experts with LLM inference to detect adverse safety events (ASEs) with minimal errors. We tested our method in identifying medication errors in medical charts and compared results to reviews by expert physicians. Our method detected epinephrine delays and overdoses with a high level of accuracy. SAFE-AI presents a novel approach for clinical AI applications that overcomes reliance on pattern recognition instead of medical knowledge biases present in training data.
SN - 2767-3170
ER -