Assessing safety and trustworthiness of large language models in medicine
7 affiliations
- 1Department of Computer Science, University of Maryland at College Park, College Park, MD, USA
- 2Division of Intramural Research, National Library of Medicine (NLM), National Institutes of Health (NIH), Bethesda, MD, USA
- 3Department of Computer Science, University of Virginia, Charlottesville, VA, USA
- 4College of Medicine, Howard University, Washington, DC, USA
- 5Jacob Medical Center, Albert Einstein College of Medicine, New York City, NY, USA
- 6School of Medicine, Universidad de Chile, Santiago, Chile
- 7Department of Neurology, University of Florida, Gainesville, FL, USA
Abstract
Background The remarkable capabilities of large language models make them increasingly compelling for use in real-world healthcare applications. However, the risks associated with using these artificial intelligence systems in medicine are not systematically understood. The aim of this study is to characterize these risks by applying five key principles for safe and trustworthy medical artificial intelligence: truthfulness, resilience, fairness, robustness, and privacy. Methods We introduce MedGuard-Bench: a safety benchmark featuring one thousand expert-verified questions covering ten specific aspects of our five core principles. We use this comprehensive corpus to systematically evaluate sixteen commonly used large language models, assessing their safety and reliability in medical contexts. Results We show that current large language models generally perform poorly on most of our safety tests, regardless of their safety alignment mechanisms. Our evaluation demonstrates that these models fall significantly short when compared to the high performance and reliability of human physicians. Conclusions Despite reports indicating that advanced large language models can match or exceed human performance in various medical tasks, this study reveals a significant safety gap in current technology. This underscores the crucial need for ongoing human oversight and the implementation of strict safety guardrails before deploying these tools in clinical practice.
Related
- A Survey on Medical Large Language Models: Technology, Application, Trustworthiness, and Future Directions
- Clinical performance and practical limitations of large language models in respiratory medicine
- Adjudicator sensitivity of safety estimates in large language models for open-ended cardiovascular reasoning: a blinded multi-adjudicator benchmark study running title: adjudicator variability in LLM safety
- Evaluating a Large Language Model for Emergency Triage Using Synthetic Clinical Dialogues: Toward a Multimodal Clinical Safety Framework
- How well do large language models answer postoperative lumbar fusion questions? A blinded comparative analysis of accuracy, quality, readability, and safety-related content
BibTeX
@article{yang2026assessing,
title = {{Assessing safety and trustworthiness of large language models in medicine}},
author = {Yifan Yang and Qiao Jin and Robert Leaman and Xiaoyu Liu and Guangzhi Xiong and Maame Sarfo-Gyamfi and Changlin Gong and Santiago Ferrière-Steinert and W. John Wilbur and Xiaojun Li and Jiaxin Yuan and Bang An and Kelvin Castro Neira and Francisco Erramuspe Álvarez and Matías Stockle and Aidong Zhang and Furong Huang and Zhiyong Lu},
journal = {Communications Medicine},
year = {2026},
month = sep,
doi = {10.1038/s43856-026-01881-3},
url = {https://doi.org/10.1038/s43856-026-01881-3},
note = {CC BY},
}
RIS
TY - JOUR
TI - Assessing safety and trustworthiness of large language models in medicine
AU - Yifan Yang
AU - Qiao Jin
AU - Robert Leaman
AU - Xiaoyu Liu
AU - Guangzhi Xiong
AU - Maame Sarfo-Gyamfi
AU - Changlin Gong
AU - Santiago Ferrière-Steinert
AU - W. John Wilbur
AU - Xiaojun Li
AU - Jiaxin Yuan
AU - Bang An
AU - Kelvin Castro Neira
AU - Francisco Erramuspe Álvarez
AU - Matías Stockle
AU - Aidong Zhang
AU - Furong Huang
AU - Zhiyong Lu
PY - 2026
DA - 2026/09/19
JO - Communications Medicine
DO - 10.1038/s43856-026-01881-3
UR - https://doi.org/10.1038/s43856-026-01881-3
AB - Background The remarkable capabilities of large language models make them increasingly compelling for use in real-world healthcare applications. However, the risks associated with using these artificial intelligence systems in medicine are not systematically understood. The aim of this study is to characterize these risks by applying five key principles for safe and trustworthy medical artificial intelligence: truthfulness, resilience, fairness, robustness, and privacy. Methods We introduce MedGuard-Bench: a safety benchmark featuring one thousand expert-verified questions covering ten specific aspects of our five core principles. We use this comprehensive corpus to systematically evaluate sixteen commonly used large language models, assessing their safety and reliability in medical contexts. Results We show that current large language models generally perform poorly on most of our safety tests, regardless of their safety alignment mechanisms. Our evaluation demonstrates that these models fall significantly short when compared to the high performance and reliability of human physicians. Conclusions Despite reports indicating that advanced large language models can match or exceed human performance in various medical tasks, this study reveals a significant safety gap in current technology. This underscores the crucial need for ongoing human oversight and the implementation of strict safety guardrails before deploying these tools in clinical practice.
SN - 2730-664X
ER -