Accuracy and confidence calibration of large language models on neurology board-style questions
- 1Imperial College London
Abstract
Large language models (LLMs) are increasingly evaluated for clinical tasks, but safe use requires reliable uncertainty estimates as well as high accuracy. We tested eight general-purpose LLMs on 100 validated neurology board-style multiple-choice questions and compared three confidence signals: self-reported confidence, critique-implied confidence from an independent model instance, and consistency-based confidence from repeated sampling. Accuracy varied from 45% (Llama 3.1 8B) to 92% (Gemini 3 Flash/Pro), but confidence was generally overestimated, particularly in lower-accuracy models. Consistency-based confidence provided the best discrimination between correct and incorrect answers (pooled AUROC 0.74), outperforming self-report, although calibration differences between methods were not significant. Calibration should therefore be reported alongside accuracy when clinical LLMs are evaluated.
Short DescriptionWe evaluated the accuracy and confidence of eight large language models on 100 neurology board-style questions. We found substantial overconfidence, and differences between confidence methods in their ability to distinguish correct from incorrect answers.
Related
- How well do large language models answer postoperative lumbar fusion questions? A blinded comparative analysis of accuracy, quality, readability, and safety-related content
- Non-generative readout with few-label calibration versus LoRA fine-tuning of a large language model for ASA physical status classification: accuracy, speed, cost and label efficiency in three data sources
- A large language model for risk-of-bias assessment in systematic reviews of prognosis studies in clinical neurology
- Safety and guideline concordance of public large language model responses to patient-facing questions on active charcot neuro-osteoarthropathy
- Comparison of the performance of large language models (ChatGPT-5.5, Gemini 3.1, and Claude 4.6) in answering pharmacology questions
BibTeX
@misc{graham2026accuracy,
title = {{Accuracy and confidence calibration of large language models on neurology board-style questions}},
author = {N. S. Graham and S. D. Auger and B. Attaallah and A. Batla and G. Scott},
howpublished = {medRxiv},
year = {2026},
month = oct,
doi = {10.64898/2026.10.17.26365807},
url = {https://www.medrxiv.org/content/10.64898/2026.10.17.26365807v1},
note = {CC BY},
}
RIS
TY - UNPB
TI - Accuracy and confidence calibration of large language models on neurology board-style questions
AU - N. S. Graham
AU - S. D. Auger
AU - B. Attaallah
AU - A. Batla
AU - G. Scott
PY - 2026
DA - 2026/10/17
PB - medRxiv
DO - 10.64898/2026.10.17.26365807
UR - https://www.medrxiv.org/content/10.64898/2026.10.17.26365807v1
AB - Large language models (LLMs) are increasingly evaluated for clinical tasks, but safe use requires reliable uncertainty estimates as well as high accuracy. We tested eight general-purpose LLMs on 100 validated neurology board-style multiple-choice questions and compared three confidence signals: self-reported confidence, critique-implied confidence from an independent model instance, and consistency-based confidence from repeated sampling. Accuracy varied from 45% (Llama 3.1 8B) to 92% (Gemini 3 Flash/Pro), but confidence was generally overestimated, particularly in lower-accuracy models. Consistency-based confidence provided the best discrimination between correct and incorrect answers (pooled AUROC 0.74), outperforming self-report, although calibration differences between methods were not significant. Calibration should therefore be reported alongside accuracy when clinical LLMs are evaluated. Short DescriptionWe evaluated the accuracy and confidence of eight large language models on 100 neurology board-style questions. We found substantial overconfidence, and differences between confidence methods in their ability to distinguish correct from incorrect answers.
ER -