Evaluating Large Language Models on Clinical Risk Judgement: An Example from Alcohol Use Disorder
Abstract
Large language models (LLMs) are being explored for medical risk assessment, yet performance on standardized benchmarks does not necessarily establish clinical competency. The present study evaluated whether LLM-generated Alcohol Use Disorder (AUD) risk judgments aligned with epidemiological evidence and remained consistent as clinical and demographic context changed. Three LLMs—GPT-5.5, LLaMA-2-70B, and MEDITRON-70B—completed four pairwise-comparison experiments involving DSM-5 AUD criteria, demographic characteristics, criteria paired with demographic characteristics, and narrative clinical vignettes. LLM-implied rankings were compared with epidemiological benchmarks derived from the National Epidemiological Survey on Alcohol and Related Conditions, Wave Three (NESARC-III). Our results show that none of the LLMs reproduced the implied severity ordering of the AUD criteria. Demographic cues consistently influenced AUD risk judgments, although the direction and magnitude of these effects varied across LLMs, demographic attributes, and clinical contexts. Demographic rankings also showed only partial stability as contextual information changed. LLMs differed substantially in decisiveness, with GPT-5.5 assigning substantially higher probability to selected responses than LLaMA-2-70B or MEDITRON-70B. These findings demonstrate a gap in current LLM risk-assessment capabilities and highlight the importance of evaluating the direction, magnitude, stability, and certainty of LLM judgments alongside accuracy.
Related
- Evaluating a Large Language Model for Emergency Triage Using Synthetic Clinical Dialogues: Toward a Multimodal Clinical Safety Framework
- Reply to: "Evaluating the Clinical Readiness of General-Purpose Large Language Models in Evidence Synthesis".
- Physicians and artificial intelligence diverge in evaluating large language models on real clinical cases
- Evaluating Open-Weight Large Language Models for Structured Depression Assessment from Clinical Interviews
- Evaluating and leveraging large language models in clinical pharmacology and therapeutics assessment: From exam takers to exam shapers
BibTeX
@misc{bolton2026evaluating,
title = {{Evaluating Large Language Models on Clinical Risk Judgement: An Example from Alcohol Use Disorder}},
author = {Charles M. Bolton and Kiwon Song and Katelyn A Wasson and Clintin Davis-Stober and Konstantina Sokratous},
howpublished = {PsyArXiv},
year = {2026},
month = oct,
doi = {10.31234/osf.io/msp9f_v1},
url = {https://osf.io/preprints/psyarxiv/msp9f_v1/},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Evaluating Large Language Models on Clinical Risk Judgement: An Example from Alcohol Use Disorder
AU - Charles M. Bolton
AU - Kiwon Song
AU - Katelyn A Wasson
AU - Clintin Davis-Stober
AU - Konstantina Sokratous
PY - 2026
DA - 2026/10/07
PB - PsyArXiv
DO - 10.31234/osf.io/msp9f_v1
UR - https://osf.io/preprints/psyarxiv/msp9f_v1/
AB - Large language models (LLMs) are being explored for medical risk assessment, yet performance on standardized benchmarks does not necessarily establish clinical competency. The present study evaluated whether LLM-generated Alcohol Use Disorder (AUD) risk judgments aligned with epidemiological evidence and remained consistent as clinical and demographic context changed. Three LLMs—GPT-5.5, LLaMA-2-70B, and MEDITRON-70B—completed four pairwise-comparison experiments involving DSM-5 AUD criteria, demographic characteristics, criteria paired with demographic characteristics, and narrative clinical vignettes. LLM-implied rankings were compared with epidemiological benchmarks derived from the National Epidemiological Survey on Alcohol and Related Conditions, Wave Three (NESARC-III). Our results show that none of the LLMs reproduced the implied severity ordering of the AUD criteria. Demographic cues consistently influenced AUD risk judgments, although the direction and magnitude of these effects varied across LLMs, demographic attributes, and clinical contexts. Demographic rankings also showed only partial stability as contextual information changed. LLMs differed substantially in decisiveness, with GPT-5.5 assigning substantially higher probability to selected responses than LLaMA-2-70B or MEDITRON-70B. These findings demonstrate a gap in current LLM risk-assessment capabilities and highlight the importance of evaluating the direction, magnitude, stability, and certainty of LLM judgments alongside accuracy.
ER -