Response Quality of AI-Generated Answers to Orthodontic Patient Concerns Across Empathy, Accuracy, Comprehensiveness, Personalisation, Safety, and Patient-Centeredness: A Cross-Model, Bilingual Evaluation of Claude, GPT-4o, and Gemini.
- 1Department of Orthodontics, Faculty of Dentistry, Istanbul Health and Technology University, Istanbul 34445, Turkey.
Abstract
Background
Patients now consult large language models (LLMs) for orthodontic concerns, but most evaluations sit in English and focus on factual accuracy. We compared three current LLMs (Claude Opus 4.5, GPT-4o, Gemini 2.5 Flash) on patient-facing responses in English and Turkish, and tested whether model differences held up after adjusting for response length.Methods
Sixty standardised patient scenarios were prompted in both languages to each model via stateless single-turn API calls with identical token limits, yielding 360 responses. Three blinded orthodontists scored every response on empathy, accuracy, comprehensiveness, personalisation, safety, and patient-centeredness (1-5 Likert). Primary between-model contrasts used mixed-effects linear models with scenario as a random effect to accommodate the paired scenario structure, with length-adjusted specifications adding response word count as a covariate; one-way ANOVA and ANCOVA on the balanced English sample were reported concordantly as descriptive checks. Reliability was assessed with ICC(2,3) and Krippendorff's α.Results
Forty-eight Turkish Gemini outputs (80%) were truncated mid-generation and were excluded, leaving 312 successfully completed responses. Between-model ranking is therefore reported on the balanced English cell (n = 60 per model). Unadjusted, Gemini scored highest on the English cell (total 4.04 vs. 3.86 GPT-4o, 3.85 Claude). But response length correlated strongly with scores (r = 0.75 total; r = 0.84 comprehensiveness), and mean Gemini English length (547 words) was roughly double Claude's (265) and GPT-4o's (288). After length adjustment, Gemini's advantage on empathy, comprehensiveness, and personalisation disappeared (all p > 0.10), the total-score coefficient reversed (β = -0.065, p = 0.017), and Gemini scored lower on patient-centeredness (β = -0.153, p = 0.021). English scored substantially higher than Turkish on every dimension (3.92 vs. 3.51; d = 2.71), and patient-centeredness was the weakest dimension overall (3.27).Conclusions
Rubric-based scoring of LLM outputs is strongly associated with response length; unadjusted between-model comparisons may mainly reflect length differences rather than architectural differences. The English-Turkish gap and the Turkish Gemini truncation are robust to length adjustment. Per-language quality assurance-for both completeness and length-normalised content quality-is a prerequisite for patient-facing LLM deployment; English-only evaluation is not a safe proxy.
Related
- Evaluation of the effects of Er,Cr:YSGG laser irradiation on enamel demineralization and orthodontic shear bond strength: an in vitro comparative study with topical fluoride
- A Comparative Evaluation of the Effects of Three Different Mouthwashes on Salivary pH in Patients Undergoing Fixed Orthodontic Treatment: A Randomized Clinical Trial.
- Citation Fabrication and Accuracy in AI-Generated Psychiatric Research: A Comparative Evaluation of Four Contemporary Large Language Models
- Nutrient Calculation Accuracy, Expert-Rated Clinical Appropriateness and Safety Indicators, and Reproducibility of Large Language Model-Generated Diet Plans for Polyendocrine Metabolic Ovarian Syndrome: A Real-Case-Based Evaluation
- Application of spectroscopic and multispectral imaging technologies on the assessment of ready-to-eat pineapple quality: A performance evaluation study of machine learning models generated from two commercial data analytics tools
BibTeX
@article{ozcan2026response,
title = {{Response Quality of AI-Generated Answers to Orthodontic Patient Concerns Across Empathy, Accuracy, Comprehensiveness, Personalisation, Safety, and Patient-Centeredness: A Cross-Model, Bilingual Evaluation of Claude, GPT-4o, and Gemini.}},
author = {Mustafa Ozcan and Ferdi Allaf},
journal = {Healthcare (Basel, Switzerland)},
year = {2026},
month = oct,
doi = {10.3390/healthcare14193281},
url = {https://europepmc.org/article/MED/42881719},
note = {CC BY},
}
RIS
TY - JOUR
TI - Response Quality of AI-Generated Answers to Orthodontic Patient Concerns Across Empathy, Accuracy, Comprehensiveness, Personalisation, Safety, and Patient-Centeredness: A Cross-Model, Bilingual Evaluation of Claude, GPT-4o, and Gemini.
AU - Mustafa Ozcan
AU - Ferdi Allaf
PY - 2026
DA - 2026/10/02
JO - Healthcare (Basel, Switzerland)
DO - 10.3390/healthcare14193281
UR - https://europepmc.org/article/MED/42881719
AB - Background Patients now consult large language models (LLMs) for orthodontic concerns, but most evaluations sit in English and focus on factual accuracy. We compared three current LLMs (Claude Opus 4.5, GPT-4o, Gemini 2.5 Flash) on patient-facing responses in English and Turkish, and tested whether model differences held up after adjusting for response length.Methods Sixty standardised patient scenarios were prompted in both languages to each model via stateless single-turn API calls with identical token limits, yielding 360 responses. Three blinded orthodontists scored every response on empathy, accuracy, comprehensiveness, personalisation, safety, and patient-centeredness (1-5 Likert). Primary between-model contrasts used mixed-effects linear models with scenario as a random effect to accommodate the paired scenario structure, with length-adjusted specifications adding response word count as a covariate; one-way ANOVA and ANCOVA on the balanced English sample were reported concordantly as descriptive checks. Reliability was assessed with ICC(2,3) and Krippendorff's α.Results Forty-eight Turkish Gemini outputs (80%) were truncated mid-generation and were excluded, leaving 312 successfully completed responses. Between-model ranking is therefore reported on the balanced English cell (n = 60 per model). Unadjusted, Gemini scored highest on the English cell (total 4.04 vs. 3.86 GPT-4o, 3.85 Claude). But response length correlated strongly with scores (r = 0.75 total; r = 0.84 comprehensiveness), and mean Gemini English length (547 words) was roughly double Claude's (265) and GPT-4o's (288). After length adjustment, Gemini's advantage on empathy, comprehensiveness, and personalisation disappeared (all p > 0.10), the total-score coefficient reversed (β = -0.065, p = 0.017), and Gemini scored lower on patient-centeredness (β = -0.153, p = 0.021). English scored substantially higher than Turkish on every dimension (3.92 vs. 3.51; d = 2.71), and patient-centeredness was the weakest dimension overall (3.27).Conclusions Rubric-based scoring of LLM outputs is strongly associated with response length; unadjusted between-model comparisons may mainly reflect length differences rather than architectural differences. The English-Turkish gap and the Turkish Gemini truncation are robust to length adjustment. Per-language quality assurance-for both completeness and length-normalised content quality-is a prerequisite for patient-facing LLM deployment; English-only evaluation is not a safe proxy.
SN - 2227-9032
ER -