Clinical accuracy, information quality and readability of chatbot responses to patient questions about lower extremity amputation
Abstract
Objective To compare the clinical accuracy, information quality and readability of chatbot responses to common patient questions about lower extremity amputation. Methods Twenty frequently asked patient questions were submitted once to each of ChatGPT, Gemini and DeepSeek in separate sessions. Two reviewers, blinded to chatbot identity, independently assessed responses using a four-category clinical accuracy scale and the DISCERN instrument. Readability was quantified as the mean grade level across the Flesch-Kincaid Grade Level, Gunning Fog, Coleman-Liau, and Simple Measure of Gobbledygook indices. Comparisons used Friedman tests and Holm-adjusted Wilcoxon signed-rank tests. Results Sixty responses were analysed. Mean (±standard deviation) clinical accuracy scores were 2.20 ± 0.57, 1.55 ± 0.65 and 1.20 ± 0.38 for ChatGPT, Gemini and DeepSeek, respectively (lower scores indicate greater accuracy). Gemini and DeepSeek received lower scores than ChatGPT (adjusted p=0.012 and p=0.002, respectively), whereas the Gemini–DeepSeek difference was not significant (p=0.090). Mean DISCERN scores were 50.05 ± 5.78, 53.33 ± 5.43 and 58.23 ± 3.39, respectively, with significant differences in all pairwise comparisons. Mean reading grades were 12.93 ± 1.28, 13.17 ± 1.36 and 11.74 ± 1.15, respectively; DeepSeek had lower grades than both comparators (both adjusted p<0.001). Conclusion DeepSeek responses received higher information quality scores and had lower reading grade levels, although clinical accuracy did not differ significantly between DeepSeek and Gemini. All three chatbots exceeded commonly recommended reading levels for patient education. Clinician review and language adaptation are warranted before chatbot-generated information is used to supplement individualised counselling on amputation and rehabilitation.
Related
- Quality, Readability, Understandability, and Actionability of AI Chatbots’ Responses to Dental Patient Questions: A Comparative Evaluation
- Quality and readability of responses generated by popular large language models to frequently asked questions about menopause: A comparative study
- Quality and Readability of ChatGPT and Gemini Responses to Patient Questions About Systemic Isotretinoin: Comparative Evaluation Study.
- Comparison of ChatGPT, Gemini, and DeepSeek responses to questions on medication-related osteonecrosis of the jaw (MRONJ): a comparative analysis of expert-rated clinical quality, readability, response time, and clinical safety risk.
- How well do large language models answer postoperative lumbar fusion questions? A blinded comparative analysis of accuracy, quality, readability, and safety-related content
BibTeX
@article{balaban2026clinical,
title = {{Clinical accuracy, information quality and readability of chatbot responses to patient questions about lower extremity amputation}},
author = {Kamil Balaban MD and Mehmet Batu Ertan MD and Mahmut Kalem MD},
journal = {Journal of Orthopaedic Surgery},
year = {2026},
month = nov,
doi = {10.1177/10225536261504909},
url = {https://doi.org/10.1177/10225536261504909},
note = {CC BY},
}
RIS
TY - JOUR
TI - Clinical accuracy, information quality and readability of chatbot responses to patient questions about lower extremity amputation
AU - Kamil Balaban MD
AU - Mehmet Batu Ertan MD
AU - Mahmut Kalem MD
PY - 2026
DA - 2026/11/05
JO - Journal of Orthopaedic Surgery
DO - 10.1177/10225536261504909
UR - https://doi.org/10.1177/10225536261504909
AB - Objective To compare the clinical accuracy, information quality and readability of chatbot responses to common patient questions about lower extremity amputation. Methods Twenty frequently asked patient questions were submitted once to each of ChatGPT, Gemini and DeepSeek in separate sessions. Two reviewers, blinded to chatbot identity, independently assessed responses using a four-category clinical accuracy scale and the DISCERN instrument. Readability was quantified as the mean grade level across the Flesch-Kincaid Grade Level, Gunning Fog, Coleman-Liau, and Simple Measure of Gobbledygook indices. Comparisons used Friedman tests and Holm-adjusted Wilcoxon signed-rank tests. Results Sixty responses were analysed. Mean (±standard deviation) clinical accuracy scores were 2.20 ± 0.57, 1.55 ± 0.65 and 1.20 ± 0.38 for ChatGPT, Gemini and DeepSeek, respectively (lower scores indicate greater accuracy). Gemini and DeepSeek received lower scores than ChatGPT (adjusted p=0.012 and p=0.002, respectively), whereas the Gemini–DeepSeek difference was not significant (p=0.090). Mean DISCERN scores were 50.05 ± 5.78, 53.33 ± 5.43 and 58.23 ± 3.39, respectively, with significant differences in all pairwise comparisons. Mean reading grades were 12.93 ± 1.28, 13.17 ± 1.36 and 11.74 ± 1.15, respectively; DeepSeek had lower grades than both comparators (both adjusted p<0.001). Conclusion DeepSeek responses received higher information quality scores and had lower reading grade levels, although clinical accuracy did not differ significantly between DeepSeek and Gemini. All three chatbots exceeded commonly recommended reading levels for patient education. Clinician review and language adaptation are warranted before chatbot-generated information is used to supplement individualised counselling on amputation and rehabilitation.
SN - 2309-4990
ER -