Evaluating large language models in patient education: a comparative analysis addressing frequently asked questions in peri-acetabular osteotomy
- 1University College London Hospitals NHS Foundation Trust, UK
Abstract
Introduction: Large language models (LLMs) are increasingly used as sources of information across many fields, including healthcare. As patients turn to these models for health-related queries, evaluating the accuracy and reliability of their responses is essential. Peri-acetabular osteotomy (PAO) is performed on younger patients – a group more likely to use digital tools like LLMs for health information. This study assesses the accuracy and readability performance of two leading LLMs, ChatGPT and Google Gemini, in addressing common patient questions on PAO.
Methods: A panel of fellowship-trained PAO surgeons created ten commonly asked patient questions based on real-world experience. Responses from each LLM were assessed by the same three surgeons, blinded to response origin, using a 5-point Likert scale to evaluate clarity, accuracy, and completeness. Readability was measured with Flesch–Kincaid Reading scores.
Results: ChatGPT outperformed Gemini with an average score of 4.17 vs 3.13 (t = −3.08, p = 0.006). ChatGPT’s responses were often rated higher for completeness and clarity, particularly in areas needing detailed explanation, and usually required minimal clarification. Gemini sometimes lacked specificity or included minor inaccuracies that reduced its perceived reliability. Both LLMs produced responses with similar “difficult” Flesch Reading Ease scores.
Conclusions: There may be significant differences in how effectively LLMs support patients with surgical queries. ChatGPT more consistently met expert standards for clarity and thoroughness. As LLM usage expands, ChatGPT may aid patient education on hip surgery, supporting consultations, informed decisions and postoperative guidance.
Related
- Pelvic Obliquity Impacts Acetabular Coverage Measurements, and the Elevated Hip Is More Frequently Symptomatic in Hip Preservation Patients
- Evaluating the reliability, quality, and readability of AI-generated patient education on hallux valgus: a comparative study of large language models
- Evaluating the Performance of Five Leading Large Language Models in Generating Health Education Materials for Patients With Infertility: Comparative Analysis.
- Higher Education Institutions: An Underused Setting for Addressing Adverse Childhood Experiences.
- Quality and readability of responses generated by popular large language models to frequently asked questions about menopause: A comparative study
BibTeX
@article{davis2026evaluating,
title = {{Evaluating large language models in patient education: a comparative analysis addressing frequently asked questions in peri-acetabular osteotomy}},
author = {TP Davis and B Guevel and K Logishetty and AG Dick and J Hutt},
journal = {Annals of The Royal College of Surgeons of England},
year = {2026},
month = aug,
doi = {10.1308/rcsann.2026.0056},
url = {https://doi.org/10.1308/rcsann.2026.0056},
note = {CC BY 4.0},
}
RIS
TY - JOUR
TI - Evaluating large language models in patient education: a comparative analysis addressing frequently asked questions in peri-acetabular osteotomy
AU - TP Davis
AU - B Guevel
AU - K Logishetty
AU - AG Dick
AU - J Hutt
PY - 2026
DA - 2026/08/27
JO - Annals of The Royal College of Surgeons of England
DO - 10.1308/rcsann.2026.0056
UR - https://doi.org/10.1308/rcsann.2026.0056
AB - Introduction: Large language models (LLMs) are increasingly used as sources of information across many fields, including healthcare. As patients turn to these models for health-related queries, evaluating the accuracy and reliability of their responses is essential. Peri-acetabular osteotomy (PAO) is performed on younger patients – a group more likely to use digital tools like LLMs for health information. This study assesses the accuracy and readability performance of two leading LLMs, ChatGPT and Google Gemini, in addressing common patient questions on PAO. Methods: A panel of fellowship-trained PAO surgeons created ten commonly asked patient questions based on real-world experience. Responses from each LLM were assessed by the same three surgeons, blinded to response origin, using a 5-point Likert scale to evaluate clarity, accuracy, and completeness. Readability was measured with Flesch–Kincaid Reading scores. Results: ChatGPT outperformed Gemini with an average score of 4.17 vs 3.13 (t = −3.08, p = 0.006). ChatGPT’s responses were often rated higher for completeness and clarity, particularly in areas needing detailed explanation, and usually required minimal clarification. Gemini sometimes lacked specificity or included minor inaccuracies that reduced its perceived reliability. Both LLMs produced responses with similar “difficult” Flesch Reading Ease scores. Conclusions: There may be significant differences in how effectively LLMs support patients with surgical queries. ChatGPT more consistently met expert standards for clarity and thoroughness. As LLM usage expands, ChatGPT may aid patient education on hip surgery, supporting consultations, informed decisions and postoperative guidance.
SN - 0035-8843
SN - 1478-7083
ER -