Evaluator-Masked Assessment of 4 Consumer-Facing Large Language Models on Expert Anterior Segment Ophthalmology Questions.
- 1Department of Ophthalmology, Eunpyeong St. Mary's Hospital, College of Medicine, The Catholic University of Korea, Seoul, Republic of Korea.
Abstract
Objective
To compare expert-rated quality and safety of responses from four contemporary large language models (LLMs)-ChatGPT, Gemini, Claude, and Grok-on expert-level anterior segment ophthalmology questions.Design
Cross-sectional, masked, randomized-display-order, multievaluator comparative study.Participants
Eleven fellowship-trained cornea and anterior segment subspecialists.Methods
Forty-five expert-level questions spanning six anterior segment domains were administered to each LLM under zero-shot conditions in a single prespecified session (January 19, 2026), with memory, browsing, and retrieval disabled. Responses were masked to model identity and independently rated on five 5-point Likert metrics (expert-rated accuracy, completeness/clinical utility, clarity/readability, overall safety, and potential for harm). No specialist-authored reference answer key was used; all metrics index expert-rated perception rather than verified clinical correctness.Main outcome measures
Median [interquartile range], mean ± standard deviation, favorable-rating proportion (≤2) with 95% bootstrap confidence interval, Friedman omnibus with Kendall's W, and Bonferroni-adjusted pairwise Wilcoxon tests with rank-biserial correlation. Prespecified sensitivity analyses included length-adjusted mixed models and a masking-integrity diagnostic; post hoc analyses added question-level severe-safety and style-restricted comparisons.Results
All 9900 ratings (45 questions × 4 LLMs × 5 metrics × 11 evaluators) were complete. Differences were significant for all three quality metrics (Friedman P < 0.001; Kendall's W = 0.44-0.54); Claude and Grok were each rated more favorably than ChatGPT and Gemini (all adjusted P < 0.001; |r_rb| = 0.73-0.98). No statistically significant between-model difference was detected in mean safety scores. Post hoc, more questions elicited ≥ 1 severe overall-safety rating for Claude (35.6%) and Grok (33.3%) than for ChatGPT (6.7%) or Gemini (11.1%) (Cochran Q = 23.4, P < 0.001), persisting when two-evaluator concurrence was required. Ordinal Krippendorff's α was 0.021-0.168; the 11-evaluator mean used as the primary analytic unit showed moderate reliability (intraclass correlation coefficient [ICC(2,k)] = 0.58-0.69 for quality metrics).Conclusions
Claude and Grok received more favorable expert-rated quality than ChatGPT and Gemini. Post hoc, a larger severe-safety-rating tail was identified for these two models despite no significant difference in mean safety; this signal is hypothesis-generating. The findings reflect expert preference, not validated clinical correctness, at one time point.Financial disclosures
The authors have no proprietary or commercial interest in any materials discussed in this article.
Related
- RoB2-Evaluator: Development and Technical Evaluation of a Rule-Constrained Large Language Model System for Cochrane Risk of Bias 2 Assessment
- SIGHT: A Methodological Appraisal Framework for Evaluating Large Language Models in Ophthalmology
- How well do large language models answer postoperative lumbar fusion questions? A blinded comparative analysis of accuracy, quality, readability, and safety-related content
- Safety and guideline concordance of public large language model responses to patient-facing questions on active charcot neuro-osteoarthropathy
- Comparison of the performance of large language models (ChatGPT-5.5, Gemini 3.1, and Claude 4.6) in answering pharmacology questions
BibTeX
@article{yang2026evaluatormasked,
title = {{Evaluator-Masked Assessment of 4 Consumer-Facing Large Language Models on Expert Anterior Segment Ophthalmology Questions.}},
author = {Soonwon Yang and Hyoji Han and Hyun Soo Lee},
journal = {Ophthalmology science},
year = {2026},
month = sep,
doi = {10.1016/j.xops.2026.101398},
url = {https://europepmc.org/article/MED/42894879},
note = {CC BY-NC-ND},
}
RIS
TY - JOUR
TI - Evaluator-Masked Assessment of 4 Consumer-Facing Large Language Models on Expert Anterior Segment Ophthalmology Questions.
AU - Soonwon Yang
AU - Hyoji Han
AU - Hyun Soo Lee
PY - 2026
DA - 2026/09/04
JO - Ophthalmology science
DO - 10.1016/j.xops.2026.101398
UR - https://europepmc.org/article/MED/42894879
AB - Objective To compare expert-rated quality and safety of responses from four contemporary large language models (LLMs)-ChatGPT, Gemini, Claude, and Grok-on expert-level anterior segment ophthalmology questions.Design Cross-sectional, masked, randomized-display-order, multievaluator comparative study.Participants Eleven fellowship-trained cornea and anterior segment subspecialists.Methods Forty-five expert-level questions spanning six anterior segment domains were administered to each LLM under zero-shot conditions in a single prespecified session (January 19, 2026), with memory, browsing, and retrieval disabled. Responses were masked to model identity and independently rated on five 5-point Likert metrics (expert-rated accuracy, completeness/clinical utility, clarity/readability, overall safety, and potential for harm). No specialist-authored reference answer key was used; all metrics index expert-rated perception rather than verified clinical correctness.Main outcome measures Median [interquartile range], mean ± standard deviation, favorable-rating proportion (≤2) with 95% bootstrap confidence interval, Friedman omnibus with Kendall's W, and Bonferroni-adjusted pairwise Wilcoxon tests with rank-biserial correlation. Prespecified sensitivity analyses included length-adjusted mixed models and a masking-integrity diagnostic; post hoc analyses added question-level severe-safety and style-restricted comparisons.Results All 9900 ratings (45 questions × 4 LLMs × 5 metrics × 11 evaluators) were complete. Differences were significant for all three quality metrics (Friedman P < 0.001; Kendall's W = 0.44-0.54); Claude and Grok were each rated more favorably than ChatGPT and Gemini (all adjusted P < 0.001; |r_rb| = 0.73-0.98). No statistically significant between-model difference was detected in mean safety scores. Post hoc, more questions elicited ≥ 1 severe overall-safety rating for Claude (35.6%) and Grok (33.3%) than for ChatGPT (6.7%) or Gemini (11.1%) (Cochran Q = 23.4, P < 0.001), persisting when two-evaluator concurrence was required. Ordinal Krippendorff's α was 0.021-0.168; the 11-evaluator mean used as the primary analytic unit showed moderate reliability (intraclass correlation coefficient [ICC(2,k)] = 0.58-0.69 for quality metrics).Conclusions Claude and Grok received more favorable expert-rated quality than ChatGPT and Gemini. Post hoc, a larger severe-safety-rating tail was identified for these two models despite no significant difference in mean safety; this signal is hypothesis-generating. The findings reflect expert preference, not validated clinical correctness, at one time point.Financial disclosures The authors have no proprietary or commercial interest in any materials discussed in this article.
SN - 2666-9145
ER -