Calibration of Self-Reported Confidence and Accuracy of Large Language Models in Medical Question Answering
- 1Department of Anesthesiology and Intensive Care Medicine (CCM/CVK), Charité - University Medical Center Berlin, corporate member of Freie Universität Berlin and Humboldt-Universität zu Berlin, Berlin, Germany. [email protected]
- 2Institute of Medical Informatics, Charité - University Medical Center Berlin, corporate member of Freie Universität Berlin and Humboldt-Universität zu Berlin, Charitéplatz 1, 10117, Berlin, Germany. [email protected]
- 3Institute of Medical Informatics, Charité - University Medical Center Berlin, corporate member of Freie Universität Berlin and Humboldt-Universität zu Berlin, Charitéplatz 1, 10117, Berlin, Germany
Abstract
OBJECTIVE: To evaluate the alignment between self-reported confidence of large language models (LLM) and their accuracy in answering medical multiple-choice questions. MATERIALS AND METHODS: We prompted six LLMs (GPT-5, GPT-5-mini, GPT-5-nano, GPT-4o, Claude Sonnet 4.5, and Gemini 2.5) to answer MedMCQA items and report confidence scores. Based on 12,000 LLM responses, we calculated Expected Calibration Errors (ECE) by averaging absolute differences between observed accuracy and predicted confidence. RESULTS: Mean ECE differed by model (Claude Sonnet 4.5 best: 0.06; Gpt-4o worst: 0.127) and varied across specialties ("Skin" best: 0.041; "Social & Preventive Medicine" worst: 0.141). Accuracy of examined LLMs showed analogous variation between specialties. DISCUSSION: Our results demonstrate that high accuracy does not guarantee reliable uncertainty estimation. We identified substantial heterogeneity across medical specialties, where pooled metrics masked a threefold ECE increase between best- and worst-performing domains. CONCLUSION: We recommend incorporating calibration reporting into LLM evaluations, as larger models exhibit improved "self-knowledge", but uneven overconfidence persists.
Related
- Accuracy and confidence calibration of large language models on neurology board-style questions
- Non-generative readout with few-label calibration versus LoRA fine-tuning of a large language model for ASA physical status classification: accuracy, speed, cost and label efficiency in three data sources
- Comparison of the performance of large language models (ChatGPT-5.5, Gemini 3.1, and Claude 4.6) in answering pharmacology questions
- Evidence-based clinical question answering using retrieval-augmented generation (RAG) and open-source large language models
- Performance of large language models in answering orthodontic questions: A scoping review.
BibTeX
@article{boie2026calibration,
title = {{Calibration of Self-Reported Confidence and Accuracy of Large Language Models in Medical Question Answering}},
author = {Sebastian Daniel Boie and Florian Reis and Nicolas Frey and Elias Grünewald and Felix Balzer},
journal = {Journal of Medical Systems},
year = {2026},
month = jun,
doi = {10.1007/s10916-026-02430-0},
url = {https://doi.org/10.1007/s10916-026-02430-0},
note = {CC BY},
}
RIS
TY - JOUR
TI - Calibration of Self-Reported Confidence and Accuracy of Large Language Models in Medical Question Answering
AU - Sebastian Daniel Boie
AU - Florian Reis
AU - Nicolas Frey
AU - Elias Grünewald
AU - Felix Balzer
PY - 2026
DA - 2026/06/26
JO - Journal of Medical Systems
DO - 10.1007/s10916-026-02430-0
UR - https://doi.org/10.1007/s10916-026-02430-0
AB - OBJECTIVE: To evaluate the alignment between self-reported confidence of large language models (LLM) and their accuracy in answering medical multiple-choice questions. MATERIALS AND METHODS: We prompted six LLMs (GPT-5, GPT-5-mini, GPT-5-nano, GPT-4o, Claude Sonnet 4.5, and Gemini 2.5) to answer MedMCQA items and report confidence scores. Based on 12,000 LLM responses, we calculated Expected Calibration Errors (ECE) by averaging absolute differences between observed accuracy and predicted confidence. RESULTS: Mean ECE differed by model (Claude Sonnet 4.5 best: 0.06; Gpt-4o worst: 0.127) and varied across specialties ("Skin" best: 0.041; "Social & Preventive Medicine" worst: 0.141). Accuracy of examined LLMs showed analogous variation between specialties. DISCUSSION: Our results demonstrate that high accuracy does not guarantee reliable uncertainty estimation. We identified substantial heterogeneity across medical specialties, where pooled metrics masked a threefold ECE increase between best- and worst-performing domains. CONCLUSION: We recommend incorporating calibration reporting into LLM evaluations, as larger models exhibit improved "self-knowledge", but uneven overconfidence persists.
SN - 0148-5598
SN - 1573-689X
ER -