Expert-referenced evaluation of large language models for primary suspect drug identification in the FAERS database
Abstract
BackgroundAccurate identification of primary suspect drugs in adverse event reports is a critical component of post-marketing pharmacovigilance and serves as a foundation for downstream signal detection activities. However, causal attribution in spontaneous reporting systems such as the U.S. Food and Drug Administration’s Adverse Event Reporting System (FAERS) is inherently challenging. Traditional expert-based assessments are resource-intensive and difficult to scale, highlighting the need for complementary, reproducible approaches. This study aimed to evaluate the performance of large language models (LLMs) in identifying primary suspect drugs in adverse event reports against an expert reference standard.MethodsFifty FAERS case reports were randomly selected for evaluation. Three clinical pharmacy experts established the reference standard by adjudicating the primary suspect drug(s) for each case. LLM-based evaluations were conducted using multiple prompting strategies and model configurations. The performance of LLMs, pharmacy student evaluators, and original FAERS primary suspect designations (ROLE_COD) was assessed by calculating hit rates against the expert reference standard.ResultsThe original FAERS designations demonstrated limited accuracy against the expert reference standard, with a hit rate of 0.778. In contrast, LLM-based assessments achieved consistently high hit rates ranging from 0.867 to 0.920 across diverse prompting strategies and model configurations. Student evaluators achieved a comparable hit rate of 0.872; however, LLM-based evaluations exhibited greater internal consistency and markedly shorter response times.ConclusionLLM-based evaluations demonstrated high accuracy against expert assessments and favorable efficiency relative to student evaluators. These findings highlight the potential utility of LLMs as scalable, readily accessible decision-support tools in pharmacovigilance workflows.
Related
- Multilingual evaluation of a large language model-based primary care chatbot.
- Systematic analysis of echinocandin-associated adverse reactions using the FAERS database: identification of unique DRESS and resistance signals
- Multi‐system evaluation of baseline‐referenced gray scale CNR profile deviation for phantom‐based ultrasound quality assurance
- Association between primary-site surgery and overall survival in non-metastatic and metastatic colorectal cancer: a population-based study using the Surveillance, Epidemiology, and End Results database
- A Knowledge Database of Physical Activity Interventions for Hypertension using Large Language Models
BibTeX
@article{jung2026expertreferenced,
title = {{Expert-referenced evaluation of large language models for primary suspect drug identification in the FAERS database}},
author = {Danah Jung and Chaeheon Kim and Jiyun Lee and Seonghyeon Park and Young Mi Ha and Sandy Jeong Rhie and Jae Hyun Kim and Myeong Gyu Kim},
journal = {Frontiers in Pharmacology},
year = {2026},
month = nov,
doi = {10.3389/fphar.2026.1831267},
url = {https://www.frontiersin.org/articles/10.3389/fphar.2026.1831267},
note = {CC BY 4.0},
}
RIS
TY - JOUR
TI - Expert-referenced evaluation of large language models for primary suspect drug identification in the FAERS database
AU - Danah Jung
AU - Chaeheon Kim
AU - Jiyun Lee
AU - Seonghyeon Park
AU - Young Mi Ha
AU - Sandy Jeong Rhie
AU - Jae Hyun Kim
AU - Myeong Gyu Kim
PY - 2026
DA - 2026/11/02
JO - Frontiers in Pharmacology
DO - 10.3389/fphar.2026.1831267
UR - https://www.frontiersin.org/articles/10.3389/fphar.2026.1831267
AB - BackgroundAccurate identification of primary suspect drugs in adverse event reports is a critical component of post-marketing pharmacovigilance and serves as a foundation for downstream signal detection activities. However, causal attribution in spontaneous reporting systems such as the U.S. Food and Drug Administration’s Adverse Event Reporting System (FAERS) is inherently challenging. Traditional expert-based assessments are resource-intensive and difficult to scale, highlighting the need for complementary, reproducible approaches. This study aimed to evaluate the performance of large language models (LLMs) in identifying primary suspect drugs in adverse event reports against an expert reference standard.MethodsFifty FAERS case reports were randomly selected for evaluation. Three clinical pharmacy experts established the reference standard by adjudicating the primary suspect drug(s) for each case. LLM-based evaluations were conducted using multiple prompting strategies and model configurations. The performance of LLMs, pharmacy student evaluators, and original FAERS primary suspect designations (ROLE_COD) was assessed by calculating hit rates against the expert reference standard.ResultsThe original FAERS designations demonstrated limited accuracy against the expert reference standard, with a hit rate of 0.778. In contrast, LLM-based assessments achieved consistently high hit rates ranging from 0.867 to 0.920 across diverse prompting strategies and model configurations. Student evaluators achieved a comparable hit rate of 0.872; however, LLM-based evaluations exhibited greater internal consistency and markedly shorter response times.ConclusionLLM-based evaluations demonstrated high accuracy against expert assessments and favorable efficiency relative to student evaluators. These findings highlight the potential utility of LLMs as scalable, readily accessible decision-support tools in pharmacovigilance workflows.
SN - 1663-9812
ER -