Benchmarking general and medical LLM performance in Parkinson's disease management
4 affiliations
- 1Department of Neurology and Neurosurgery, Montreal Neurological Institute, McGill University, Montreal, Canada
- 2Montreal Neurological Hospital, McGill University Centre, Montreal, QC, Canada
- 3Integrated Program in Neuroscience, McGill University, Montreal, Canada
- 4Research Institute of the McGill University Health Centre, Montreal, Canada
Abstract
Background: Large language models (LLMs) are increasingly being used as clinical support tools, yet their performance in Parkinson's disease (PD) management remains insufficiently characterized.
Methods: We evaluated a general-purpose LLM (Gemini 1.5 Pro) and a medically specialized LLM (OpenEvidence) using 45 synthetic PD vignettes requiring generative assessment and management recommendations. Responses were independently rated by two movement disorders fellows using predefined 5-point Likert scales, with adjudication by a senior specialist when required.
Results: Both models demonstrated generally good performance in clinical assessment, whereas management recommendations were less reliable. A modest numerical advantage was observed for the medically specialized model across assessment, plan, and composite outcomes, although no statistically significant differences were identified. Qualitative analysis showed that clinically relevant errors occurred in both models, predominantly reflecting treatment strategy errors.
Conclusions: These findings suggest that, in the specific model versions evaluated, LLMs may support clinical reasoning in PD, but require cautious interpretation, particularly for treatment recommendations.
Related
- Benchmarking the performance of large language models on medical genetics questions: a comparative evaluation of GPT-5.1, Gemini Pro 3, and Claude Opus 4.5.
- Charting the future: A multi-setting medical chart study of advance care planning in Parkinson's disease and atypical Parkinsonism
- Multidimensional evaluation of AI for integrated panvascular disease management: benchmarking against consensus guidelines on lipid, metabolic and antithrombotic therapy (2024–2025)
- Benchmarking Ask Eolas-a medical AI tool for antimicrobial advice available in UK hospitals
- Reforming General Academic English: a public-health-themed curriculum for Chinese medical students
BibTeX
@article{shechter2026benchmarking,
title = {{Benchmarking general and medical LLM performance in Parkinson's disease management}},
author = {Yosef Shechter and Raymond Klevor and Trycia Kouchache and Sarah Bouhadoun and Ronald B. Postuma},
journal = {Clinical Parkinsonism \& Related Disorders},
year = {2026},
month = nov,
doi = {10.1016/j.prdoa.2026.100517},
url = {https://doi.org/10.1016/j.prdoa.2026.100517},
note = {CC BY-NC-ND 4.0},
}
RIS
TY - JOUR
TI - Benchmarking general and medical LLM performance in Parkinson's disease management
AU - Yosef Shechter
AU - Raymond Klevor
AU - Trycia Kouchache
AU - Sarah Bouhadoun
AU - Ronald B. Postuma
PY - 2026
DA - 2026/11/02
JO - Clinical Parkinsonism & Related Disorders
DO - 10.1016/j.prdoa.2026.100517
UR - https://doi.org/10.1016/j.prdoa.2026.100517
AB - Background: Large language models (LLMs) are increasingly being used as clinical support tools, yet their performance in Parkinson's disease (PD) management remains insufficiently characterized. Methods: We evaluated a general-purpose LLM (Gemini 1.5 Pro) and a medically specialized LLM (OpenEvidence) using 45 synthetic PD vignettes requiring generative assessment and management recommendations. Responses were independently rated by two movement disorders fellows using predefined 5-point Likert scales, with adjudication by a senior specialist when required. Results: Both models demonstrated generally good performance in clinical assessment, whereas management recommendations were less reliable. A modest numerical advantage was observed for the medically specialized model across assessment, plan, and composite outcomes, although no statistically significant differences were identified. Qualitative analysis showed that clinically relevant errors occurred in both models, predominantly reflecting treatment strategy errors. Conclusions: These findings suggest that, in the specific model versions evaluated, LLMs may support clinical reasoning in PD, but require cautious interpretation, particularly for treatment recommendations.
SN - 2590-1125
ER -