Relationship extraction for adverse drug events in clinical notes using large language models: a retrospective cross-sectional study.
Abstract
Background
Adverse drug events (ADEs) are a critical indicator of patient safety but are often documented only in free-text clinical notes. The potential of recent advances in natural language processing (NLP), particularly generative large language models (LLMs), to identify ADEs remains understudied. This study aimed to compare the performance of multiple LLMs in identifying ADE-Drug relationships in inpatient and ambulatory clinical notes.Methods
We used clinical notes from the 2018 National NLP Clinical Challenge (n2c2) ADE dataset (inpatient; n = 505) and from outpatient encounters (n = 2555) between October 1, 2018, and December 31, 2019, at a large academic medical center based in New England. Notes were pre-processed into snippets for model input. Evaluated Models included: GPT-4o, GPT-4o-mini, Llama 3.2-3B and their instruction fine-tuned variants (including low-rank adapters for Llama). Performance was assessed using both strict and relaxed evaluations (precision, recall, and F1) for all models, followed by manual evaluation (exact semantic match, partial match, missing ADE, drug mention only, not a drug, or wrong) of the two best-performing models.Findings
Of the 1048 notes in the local test split, 450 (42.9%) contained at least one ADE. The n2c2 test set included 202 notes, of which 155 (76.7%) contained at least one ADE. GPT-4o and GPT-4o-mini were the top-performing models among those evaluated. GPT-4o consistently outperformed GPT-4o-mini in ADE extraction across both datasets, with higher F1-scores (0.524 vs. 0.381) and a more balanced precision-recall profile. Both models captured ADEs effectively in explicit and complex clinical contexts, although limitations included misclassification of pre-existing allergies and occasional conflation of therapeutic indications with adverse effects. GPT-4o achieved higher exact match coverage and fewer errors across clinical notes, indicating more reliable performance in both inpatient and ambulatory settings.Interpretation
This work establishes a foundation for integrating LLM methods into real-world drug safety surveillance, with direct implications for improving patient safety.Funding
None.
Related
- Bern Sleep-Wake Registry: temporal extraction of comorbidities from unstructured clinical text using a large language model
- Variantscape: Retrospective Study of Large Language Model-Based Extraction of Variant-Treatment-Cancer Associations from Published Oncology Abstracts
- Extraction of clinical information from faxed medical records using a small local large language model pipeline on consumer hardware
- Zero-Shot Extraction of Structured Clinical Data from Free-Text CT Reports Using Foundational Large Language Models.
- Leveraging Large Language Models for Colorectal Cancer Symptom Extraction from MIMIC-IV Clinical Notes
BibTeX
@article{plasek2026relationship,
title = {{Relationship extraction for adverse drug events in clinical notes using large language models: a retrospective cross-sectional study.}},
author = {Joseph M. Plasek and Yiming Li and Mary G. Amato and Dinah Foer and Diane L. Seger and Shayma Alzaidi and Huiyuan Zhou and Gretchen Purcell Jackson and David W. Bates and Li Zhou},
journal = {EBioMedicine},
year = {2026},
month = oct,
doi = {10.1016/j.ebiom.2026.106516},
url = {https://europepmc.org/article/MED/42892212},
note = {CC BY-NC},
}
RIS
TY - JOUR
TI - Relationship extraction for adverse drug events in clinical notes using large language models: a retrospective cross-sectional study.
AU - Joseph M. Plasek
AU - Yiming Li
AU - Mary G. Amato
AU - Dinah Foer
AU - Diane L. Seger
AU - Shayma Alzaidi
AU - Huiyuan Zhou
AU - Gretchen Purcell Jackson
AU - David W. Bates
AU - Li Zhou
PY - 2026
DA - 2026/10/17
JO - EBioMedicine
DO - 10.1016/j.ebiom.2026.106516
UR - https://europepmc.org/article/MED/42892212
AB - Background Adverse drug events (ADEs) are a critical indicator of patient safety but are often documented only in free-text clinical notes. The potential of recent advances in natural language processing (NLP), particularly generative large language models (LLMs), to identify ADEs remains understudied. This study aimed to compare the performance of multiple LLMs in identifying ADE-Drug relationships in inpatient and ambulatory clinical notes.Methods We used clinical notes from the 2018 National NLP Clinical Challenge (n2c2) ADE dataset (inpatient; n = 505) and from outpatient encounters (n = 2555) between October 1, 2018, and December 31, 2019, at a large academic medical center based in New England. Notes were pre-processed into snippets for model input. Evaluated Models included: GPT-4o, GPT-4o-mini, Llama 3.2-3B and their instruction fine-tuned variants (including low-rank adapters for Llama). Performance was assessed using both strict and relaxed evaluations (precision, recall, and F1) for all models, followed by manual evaluation (exact semantic match, partial match, missing ADE, drug mention only, not a drug, or wrong) of the two best-performing models.Findings Of the 1048 notes in the local test split, 450 (42.9%) contained at least one ADE. The n2c2 test set included 202 notes, of which 155 (76.7%) contained at least one ADE. GPT-4o and GPT-4o-mini were the top-performing models among those evaluated. GPT-4o consistently outperformed GPT-4o-mini in ADE extraction across both datasets, with higher F1-scores (0.524 vs. 0.381) and a more balanced precision-recall profile. Both models captured ADEs effectively in explicit and complex clinical contexts, although limitations included misclassification of pre-existing allergies and occasional conflation of therapeutic indications with adverse effects. GPT-4o achieved higher exact match coverage and fewer errors across clinical notes, indicating more reliable performance in both inpatient and ambulatory settings.Interpretation This work establishes a foundation for integrating LLM methods into real-world drug safety surveillance, with direct implications for improving patient safety.Funding None.
SN - 2352-3964
ER -