DiaBench: A Benchmarking Framework to Evaluate the Performance of LLMs in T2D Patient Education
- 1Stanford University, Stanford, CA, USA
Abstract
Background Effective type 2 diabetes (T2D) education improves patient outcomes, but clinician time constraints limit what can be covered during encounters. Large language models (LLMs) offer a scalable way to answer patient questions, yet their accuracy and reliability in T2D education across health literacy levels remain poorly characterized. Methods We developed DiaBench, a standardized benchmarking framework built from 96 question-and-answer pairs collected by clinicians from representative real-world patient encounters. GPT-5 transformed each pair into multiple-choice questions (MCQs) under three configurations (1-3) simulating different question formats and levels of user background. A five-member panel of certified diabetes care and education specialists (CDCESs), medical doctors (MDs), and endocrinologists validated the transformations, rating question rephrasing, answer rephrasing, and answer correctness. We then evaluated seven LLMs (GPT-51, Gemini-2.5-Flash2, Cohere Command3, Llama-44, DeepSeek-R15, Perplexity Sonar6, Mistral-Large-2.07) for accuracy and self-consistency across three runs. Results (i) Expert validation showed a consensus over MCQ transformation. For configuration 1, 98.1% (SD 0.01) of rephrased answers were appropriate, 98.1% (SD 0.01) of MCQ answers were correct, and 95.0% (SD 0.05) of rephrased questions were appropriate; corresponding values for configuration 2 were 96.3% (SD 0.05), 97.5% (SD 0.02), and 89.8% (SD 0.14). (ii) across benchmarking for configurations 1 and 2, most models achieved high accuracy ([≥]95%), but Mistral-Large-2.0, Llama 4, and Perplexity Sonar showed substantially lower self-consistency (60.4%-91.7%) than other models ([≥]95%). All models degraded in both accuracy and self-consistency on configuration 3. Conclusions DiaBench quantitatively differentiates LLM capability in T2D patient education, with Gemini, GPT-5, and DeepSeek-R1 performing strongly and other models suboptimally. DiaBench provides a transferable platform for evaluating existing and newly developed LLMs, addressing the need for standardized benchmarking before clinical deployment of LLM-based systems in patient education.
Related
- Study protocol for the Collaborative Registrar Education and Training Enhancement (CREATE) project: a mixed-methods participatory study to co-design, implement and evaluate innovations to optimise GP training.
- A multi-level evidence approach to evaluate training, performance, and system-level impact of Hamdard Force: a community mental health intervention
- Artificial intelligence in medicine: a clinician-oriented guide and operational framework to evaluate new AI devices
- Unifying Vision and Language: Benchmarking End to End Transformer Model Against the ClipCap Framework
- How to evaluate participatory action research for health promotion interventions: the pragmatic Emke framework
BibTeX
@misc{li2026diabench,
title = {{DiaBench: A Benchmarking Framework to Evaluate the Performance of LLMs in T2D Patient Education}},
author = {Y. Li and S. Syna and C. Zhang and B. Kunisetty and E. Healey and A. Scheideman and M. Shao and H. Meng and Y. K. Lin and J. Kohlenberg and Y. Zhang and D. C. Klonoff and A. Simos and M. Basina and M. Snyder and H. Chen and T. Wang},
howpublished = {medRxiv},
year = {2026},
month = oct,
doi = {10.64898/2026.10.26.26366562},
url = {https://www.medrxiv.org/content/10.64898/2026.10.26.26366562v1},
note = {CC BY-NC-ND},
}
RIS
TY - UNPB
TI - DiaBench: A Benchmarking Framework to Evaluate the Performance of LLMs in T2D Patient Education
AU - Y. Li
AU - S. Syna
AU - C. Zhang
AU - B. Kunisetty
AU - E. Healey
AU - A. Scheideman
AU - M. Shao
AU - H. Meng
AU - Y. K. Lin
AU - J. Kohlenberg
AU - Y. Zhang
AU - D. C. Klonoff
AU - A. Simos
AU - M. Basina
AU - M. Snyder
AU - H. Chen
AU - T. Wang
PY - 2026
DA - 2026/10/26
PB - medRxiv
DO - 10.64898/2026.10.26.26366562
UR - https://www.medrxiv.org/content/10.64898/2026.10.26.26366562v1
AB - Background Effective type 2 diabetes (T2D) education improves patient outcomes, but clinician time constraints limit what can be covered during encounters. Large language models (LLMs) offer a scalable way to answer patient questions, yet their accuracy and reliability in T2D education across health literacy levels remain poorly characterized. Methods We developed DiaBench, a standardized benchmarking framework built from 96 question-and-answer pairs collected by clinicians from representative real-world patient encounters. GPT-5 transformed each pair into multiple-choice questions (MCQs) under three configurations (1-3) simulating different question formats and levels of user background. A five-member panel of certified diabetes care and education specialists (CDCESs), medical doctors (MDs), and endocrinologists validated the transformations, rating question rephrasing, answer rephrasing, and answer correctness. We then evaluated seven LLMs (GPT-51, Gemini-2.5-Flash2, Cohere Command3, Llama-44, DeepSeek-R15, Perplexity Sonar6, Mistral-Large-2.07) for accuracy and self-consistency across three runs. Results (i) Expert validation showed a consensus over MCQ transformation. For configuration 1, 98.1% (SD 0.01) of rephrased answers were appropriate, 98.1% (SD 0.01) of MCQ answers were correct, and 95.0% (SD 0.05) of rephrased questions were appropriate; corresponding values for configuration 2 were 96.3% (SD 0.05), 97.5% (SD 0.02), and 89.8% (SD 0.14). (ii) across benchmarking for configurations 1 and 2, most models achieved high accuracy ([≥]95%), but Mistral-Large-2.0, Llama 4, and Perplexity Sonar showed substantially lower self-consistency (60.4%-91.7%) than other models ([≥]95%). All models degraded in both accuracy and self-consistency on configuration 3. Conclusions DiaBench quantitatively differentiates LLM capability in T2D patient education, with Gemini, GPT-5, and DeepSeek-R1 performing strongly and other models suboptimally. DiaBench provides a transferable platform for evaluating existing and newly developed LLMs, addressing the need for standardized benchmarking before clinical deployment of LLM-based systems in patient education.
ER -