Large language models for tympanostomy patient education: readability and guideline adherence
- 1Department of Otolaryngology-Head and Neck Surgery, Hackensack University Medical Center, Hackensack, NJ 07662 USA
- 2Department of Otolaryngology-Head and Neck Surgery, Rutgers New Jersey Medical School, 90 Bergen Street, Suite 8100, Newark, NJ 07103 USA
Abstract
Purpose: To evaluate the accuracy and readability of large language model (LLM)-generated patient education materials regarding tympanostomy tube placement.
Methods: Over a two-month period, ChatGPT 4o, Gemini 2.5 Flash, and Google Search AI were prompted daily using long-form and layered prompt formats covering typical concerns regarding tympanostomy. Responses were scored on a 12-point rubric adapted from the AAO-HNS Clinical Practice Guidelines (CPG), assessing diagnostic accuracy, procedural clarity, and postoperative care. Readability was evaluated using Flesch Reading Ease and Flesch-Kincaid Grade Level. For each model and prompt type, average scores and variability were analyzed with 95% confidence intervals. Between-model differences were tested with Welch’s t-tests and Cohen’s d; temporal trends were analyzed via linear regression.
Results: For long-form outputs, Google Search AI and Gemini 2.5 Flash demonstrated the highest mean CPG adherence (96.4% and 96.6%, respectively), both significantly exceeding ChatGPT 4o (84.3%; both P < .001; Cohen’s d ≈ 2.05). For layered prompt sessions, Google Search AI again demonstrated the highest adherence (91.7%), followed by Gemini 2.5 Flash (88.1%) and ChatGPT 4o (76.2%). Guideline adherence was temporally stable across all models (P > .05 for most). All outputs exceeded the recommended sixth-grade reading threshold (mean FKGL, 9.4 for long-form; 10.6 for layered), with no statistically significant readability differences between prompt strategies.
Conclusions: Google Search AI demonstrated the highest guideline concordance, though all models produced material too complex for typical patient comprehension. Structured prompting enhances clinical accuracy, but readability remains a key barrier to accessibility.
Related
- Readability of Retina Patient Education Materials Generated With a Large Language Model
- Association of Self-Reported Clinical Behavior With Guideline Adherence in Schizophrenia and Major Depressive Disorder After Guideline Education.
- Performance of five large language models in frozen shoulder patient education: a multidimensional evaluation of readability, quality, clinical alignment, and safety
- Guideline adherence and expert-rated potential safety concerns in AI-generated dietary advice for patients with cirrhosis: a comparative evaluation of large language models.
- Evaluating Large Language Model Platforms for Medical Nutrition Therapy in Autism Spectrum Disorder: Quality, Guideline Concordance, Readability, and Hallucinations
BibTeX
@article{bahethi2026large,
title = {{Large language models for tympanostomy patient education: readability and guideline adherence}},
author = {Shreeya Bahethi and Hetal Lad and Shrey Shah and Sudeepti Vedula and Brian Manzi},
journal = {European Archives of Oto-Rhino-Laryngology},
year = {2026},
month = sep,
doi = {10.1007/s00405-026-10612-2},
url = {https://doi.org/10.1007/s00405-026-10612-2},
note = {CC BY 4.0},
}
RIS
TY - JOUR
TI - Large language models for tympanostomy patient education: readability and guideline adherence
AU - Shreeya Bahethi
AU - Hetal Lad
AU - Shrey Shah
AU - Sudeepti Vedula
AU - Brian Manzi
PY - 2026
DA - 2026/09/16
JO - European Archives of Oto-Rhino-Laryngology
DO - 10.1007/s00405-026-10612-2
UR - https://doi.org/10.1007/s00405-026-10612-2
AB - Purpose: To evaluate the accuracy and readability of large language model (LLM)-generated patient education materials regarding tympanostomy tube placement. Methods: Over a two-month period, ChatGPT 4o, Gemini 2.5 Flash, and Google Search AI were prompted daily using long-form and layered prompt formats covering typical concerns regarding tympanostomy. Responses were scored on a 12-point rubric adapted from the AAO-HNS Clinical Practice Guidelines (CPG), assessing diagnostic accuracy, procedural clarity, and postoperative care. Readability was evaluated using Flesch Reading Ease and Flesch-Kincaid Grade Level. For each model and prompt type, average scores and variability were analyzed with 95% confidence intervals. Between-model differences were tested with Welch’s t-tests and Cohen’s d; temporal trends were analyzed via linear regression. Results: For long-form outputs, Google Search AI and Gemini 2.5 Flash demonstrated the highest mean CPG adherence (96.4% and 96.6%, respectively), both significantly exceeding ChatGPT 4o (84.3%; both P < .001; Cohen’s d ≈ 2.05). For layered prompt sessions, Google Search AI again demonstrated the highest adherence (91.7%), followed by Gemini 2.5 Flash (88.1%) and ChatGPT 4o (76.2%). Guideline adherence was temporally stable across all models (P > .05 for most). All outputs exceeded the recommended sixth-grade reading threshold (mean FKGL, 9.4 for long-form; 10.6 for layered), with no statistically significant readability differences between prompt strategies. Conclusions: Google Search AI demonstrated the highest guideline concordance, though all models produced material too complex for typical patient comprehension. Structured prompting enhances clinical accuracy, but readability remains a key barrier to accessibility.
SN - 0937-4477
SN - 1434-4726
ER -