Benchmarking large language models for HIV medical decision support
- 1Department of Biosystems Science and Engineering, ETH Zurich
- 2School of Life Sciences, École Polytechnique Fédérale de Lausanne
- 3Department of Infectious Diseases and Hospital Epidemiology, University Hospital Zurich
Abstract
Background Large language models (LLMs) are emerging as tools to support clinical decision making. HIV management is a compelling use case due to its complexity and dynamic nature, involving diverse treatment options, comorbidities, and adherence challenges. However, integrating LLMs into clinical practice raises concerns about accuracy, safety, and clinician acceptance. Despite growing interest, their performance in HIV care remains poorly studied, and benchmarking is lacking. Methods We developed HIVMedQA, a clinician-curated benchmark of HIV-related open-ended medical question-answer pairs spanning basic knowledge, clinical reasoning, complex patient vignettes, and bias-modified scenarios. We evaluated seven general-purpose and three medical LLMs. Performance was assessed using lexical similarity and an extended medical LLM-as-a-judge framework capturing key clinical dimensions, including question comprehension, reasoning, knowledge recall, bias, potential harm, and factual accuracy, to better capture nuances relevant to the medical domain, with additional evaluation by HIV-experienced physicians. Results Performance varies substantially across models and task complexity. Gemini 2.5 Pro achieves the highest overall scores, followed by Claude 3.5 Sonnet and MedGemma-27B. Knowledge recall is generally stronger than question comprehension or clinical reasoning. Medical LLMs do not consistently outperform general-purpose models, and model size alone does not predict performance. Several models are sensitive to cognitive bias prompts. LLM-as-a-judge scoring aligns better with clinician assessment than lexical metrics. Conclusions HIVMedQA provides a structured benchmark for evaluating LLMs in HIV clinical decision support. Current LLMs show promise, but limitations in reasoning, bias robustness, and safety indicate that careful validation, domain-specific evaluation, and clinician oversight remain essential before clinical deployment.
Related
- Extracting smoking history from clinical notes for lung cancer screening decision support: comparing a structured-judgment model with general-purpose large language models
- Large Language Models as Decision Support in Breast Cancer Management: A Comparison with Multidisciplinary Team Decision-Making
- Reliability, Quality, and Readability of Freely Available Large Language Models for Clinical Decision Support in Perforated Peptic Ulcer: A Blinded, Multidimensional Study
- Benchmarking the performance of large language models on medical genetics questions: a comparative evaluation of GPT-5.1, Gemini Pro 3, and Claude Opus 4.5.
- EM-BENCH: A Reproducible Framework for Benchmarking Guideline-Concordant Clinical Decision Making in Large Language Models
BibTeX
@article{cardenalantolin2026benchmarking,
title = {{Benchmarking large language models for HIV medical decision support}},
author = {Gonzalo Cardenal-Antolin and Jacques Fellay and Bashkim Jaha and Huldrych F. Günthard and Johannes Nemeth and Roger Kouyos and Niko Beerenwinkel and Diane Duroux},
journal = {Communications Medicine},
year = {2026},
month = aug,
doi = {10.1038/s43856-026-01875-1},
url = {https://doi.org/10.1038/s43856-026-01875-1},
note = {CC BY},
}
RIS
TY - JOUR
TI - Benchmarking large language models for HIV medical decision support
AU - Gonzalo Cardenal-Antolin
AU - Jacques Fellay
AU - Bashkim Jaha
AU - Huldrych F. Günthard
AU - Johannes Nemeth
AU - Roger Kouyos
AU - Niko Beerenwinkel
AU - Diane Duroux
PY - 2026
DA - 2026/08/01
JO - Communications Medicine
DO - 10.1038/s43856-026-01875-1
UR - https://doi.org/10.1038/s43856-026-01875-1
AB - Background Large language models (LLMs) are emerging as tools to support clinical decision making. HIV management is a compelling use case due to its complexity and dynamic nature, involving diverse treatment options, comorbidities, and adherence challenges. However, integrating LLMs into clinical practice raises concerns about accuracy, safety, and clinician acceptance. Despite growing interest, their performance in HIV care remains poorly studied, and benchmarking is lacking. Methods We developed HIVMedQA, a clinician-curated benchmark of HIV-related open-ended medical question-answer pairs spanning basic knowledge, clinical reasoning, complex patient vignettes, and bias-modified scenarios. We evaluated seven general-purpose and three medical LLMs. Performance was assessed using lexical similarity and an extended medical LLM-as-a-judge framework capturing key clinical dimensions, including question comprehension, reasoning, knowledge recall, bias, potential harm, and factual accuracy, to better capture nuances relevant to the medical domain, with additional evaluation by HIV-experienced physicians. Results Performance varies substantially across models and task complexity. Gemini 2.5 Pro achieves the highest overall scores, followed by Claude 3.5 Sonnet and MedGemma-27B. Knowledge recall is generally stronger than question comprehension or clinical reasoning. Medical LLMs do not consistently outperform general-purpose models, and model size alone does not predict performance. Several models are sensitive to cognitive bias prompts. LLM-as-a-judge scoring aligns better with clinician assessment than lexical metrics. Conclusions HIVMedQA provides a structured benchmark for evaluating LLMs in HIV clinical decision support. Current LLMs show promise, but limitations in reasoning, bias robustness, and safety indicate that careful validation, domain-specific evaluation, and clinician oversight remain essential before clinical deployment.
SN - 2730-664X
ER -