Comparative Large Language Models Evaluation for Age-Related Macular Degeneration Diagnosis on OCT and Rubric-Based Diagnostic Quality Assessment
5 affiliations
- 1Department of Biomedical Engineering, Duke University, Durham, North Carolina
- 2Department of Electrical and Computer Engineering, Duke University, Durham, North Carolina
- 3Duke Eye Center, Durham, North Carolina
- 4Department of Computer Science, Duke University, Durham, North Carolina
- 5Department of Ophthalmology, Duke University, Durham, North Carolina
Abstract
Purpose: To develop and validate a rubric-based large language model (LLM) evaluator for assessing the diagnostic quality of LLM-generated age-related macular degeneration (AMD) diagnoses from OCT scans, benchmarked against human evaluators, and, as a secondary aim, to characterize the diagnostic capability of the underlying LLMs.
Design: Retrospective study primarily developing and validating a rubric-based LLM evaluator for assessing the diagnostic quality of LLM-generated diagnoses on retinal OCT, with the diagnostic performance of the LLMs assessed as a secondary analysis.
Participants: A total of 140 OCT scans (70 of choroidal neovascular membrane, 70 of drusen), 6 diagnoser models (GPT-4.1, GPT-4o, o4-mini, Gemini-2.5-Flash, GPT-5-mini, and GPT-5.5), and 4 evaluator models (GPT-4.1, GPT-5-mini, o4-mini, and Gemini-2.5-Flash) were included in this study.
Methods: We developed a 2-stage workflow to automate OCT diagnosis associated with AMD. First, LLM “diagnosers” are prompted to provide a diagnosis in natural language from OCT scans. Second, LLM “evaluators” grade the diagnosis against a predetermined rubric for accuracy, imaging features, differential diagnosis, and clarity. A pilot experiment on 20 OCT scans benchmarked four LLM evaluators against a human evaluator using the identical rubric. Interrater reliability between the selected LLM evaluator and 2 retinal specialists was further assessed across 560 image-diagnosis pairs using intraclass correlation coefficients (ICCs) and two-1-sided test equivalence analysis. The best-performing LLM evaluator was then deployed in a full-scale analysis on 140 OCT scans to grade all 6 diagnoser models.
Main Outcome Measures: Average gradings, standard deviation, P value, correlation coefficient, ICC values, paired t-test P values, and two-1-sided test equivalence bounds between human and LLM evaluators.
Results: In the pilot analysis, Gemini-2.5-Flash demonstrated the strongest alignment with the human evaluator. In the expanded experiment across 560 image-diagnosis pairs, ICC analysis confirmed that Gemini-2.5-Flash's agreement with human specialists (ICC = 0.897) was comparable to interspecialist agreement (ICC = 0.806), and two-1-sided test analysis supported statistical equivalence with human scoring, with a mean difference of 0.123 points. In the large-scale evaluation, GPT-5.5 achieved the highest average diagnoser score (6.3/8), followed by GPT-5-mini (5.0) and Gemini-2.5-Flash (4.9).
Conclusions: Large language model evaluators demonstrate a high level of agreement with human evaluator in supervising the quality of AMD diagnoses on OCT scans generated by LLM diagnosers. This workflow demonstrates the potential of LLM-powered automation for diagnostic quality evaluation in ophthalmology clinical practice.
Financial Disclosure(s): Proprietary or commercial disclosure may be found in the Footnotes and Disclosures at the end of this article.
Related
- Diagnostic Accuracy of Pressure Injury Staging Among Nurses: A Multisite Study Using Knowledge Assessment and Image‐Based Evaluation
- Diagnostic Performance of the Radiographic Assessment of Lung Edema Score for Acute Respiratory Distress Syndrome-Post Hoc Analysis of the "Validating Acute Lung Injury Biomarkers for Diagnosis" Study.
- A comparative analysis of rapid diagnostic tests and enzyme-linked immunosorbent assays for dengue diagnosis: a retrospective study in a tertiary care hospital in India
- Diagnostic performance of a loop-mediated isothermal amplification test for the diagnosis of human and cat sporotrichosis in two highly endemic areas in Brazil: insights from a real-world, field-based evaluation.
- Laboratory evaluation of five rapid diagnostic tests for the serological diagnosis of chronic Trypanosoma cruzi infection in Yucatan, Mexico
BibTeX
@article{zhou2026comparative,
title = {{Comparative Large Language Models Evaluation for Age-Related Macular Degeneration Diagnosis on OCT and Rubric-Based Diagnostic Quality Assessment}},
author = {Ziqi Zhou and David Hunt and James M. Harris and Michael J. Fliotsos and Visrut Sudhakar and Miroslav Pajic and Majda Hadziahmetovic},
journal = {Ophthalmology Science},
year = {2026},
month = aug,
doi = {10.1016/j.xops.2026.101372},
url = {https://doi.org/10.1016/j.xops.2026.101372},
note = {CC BY-NC-ND 4.0},
}
RIS
TY - JOUR
TI - Comparative Large Language Models Evaluation for Age-Related Macular Degeneration Diagnosis on OCT and Rubric-Based Diagnostic Quality Assessment
AU - Ziqi Zhou
AU - David Hunt
AU - James M. Harris
AU - Michael J. Fliotsos
AU - Visrut Sudhakar
AU - Miroslav Pajic
AU - Majda Hadziahmetovic
PY - 2026
DA - 2026/08/26
JO - Ophthalmology Science
DO - 10.1016/j.xops.2026.101372
UR - https://doi.org/10.1016/j.xops.2026.101372
AB - Purpose: To develop and validate a rubric-based large language model (LLM) evaluator for assessing the diagnostic quality of LLM-generated age-related macular degeneration (AMD) diagnoses from OCT scans, benchmarked against human evaluators, and, as a secondary aim, to characterize the diagnostic capability of the underlying LLMs. Design: Retrospective study primarily developing and validating a rubric-based LLM evaluator for assessing the diagnostic quality of LLM-generated diagnoses on retinal OCT, with the diagnostic performance of the LLMs assessed as a secondary analysis. Participants: A total of 140 OCT scans (70 of choroidal neovascular membrane, 70 of drusen), 6 diagnoser models (GPT-4.1, GPT-4o, o4-mini, Gemini-2.5-Flash, GPT-5-mini, and GPT-5.5), and 4 evaluator models (GPT-4.1, GPT-5-mini, o4-mini, and Gemini-2.5-Flash) were included in this study. Methods: We developed a 2-stage workflow to automate OCT diagnosis associated with AMD. First, LLM “diagnosers” are prompted to provide a diagnosis in natural language from OCT scans. Second, LLM “evaluators” grade the diagnosis against a predetermined rubric for accuracy, imaging features, differential diagnosis, and clarity. A pilot experiment on 20 OCT scans benchmarked four LLM evaluators against a human evaluator using the identical rubric. Interrater reliability between the selected LLM evaluator and 2 retinal specialists was further assessed across 560 image-diagnosis pairs using intraclass correlation coefficients (ICCs) and two-1-sided test equivalence analysis. The best-performing LLM evaluator was then deployed in a full-scale analysis on 140 OCT scans to grade all 6 diagnoser models. Main Outcome Measures: Average gradings, standard deviation, P value, correlation coefficient, ICC values, paired t-test P values, and two-1-sided test equivalence bounds between human and LLM evaluators. Results: In the pilot analysis, Gemini-2.5-Flash demonstrated the strongest alignment with the human evaluator. In the expanded experiment across 560 image-diagnosis pairs, ICC analysis confirmed that Gemini-2.5-Flash's agreement with human specialists (ICC = 0.897) was comparable to interspecialist agreement (ICC = 0.806), and two-1-sided test analysis supported statistical equivalence with human scoring, with a mean difference of 0.123 points. In the large-scale evaluation, GPT-5.5 achieved the highest average diagnoser score (6.3/8), followed by GPT-5-mini (5.0) and Gemini-2.5-Flash (4.9). Conclusions: Large language model evaluators demonstrate a high level of agreement with human evaluator in supervising the quality of AMD diagnoses on OCT scans generated by LLM diagnosers. This workflow demonstrates the potential of LLM-powered automation for diagnostic quality evaluation in ophthalmology clinical practice. Financial Disclosure(s): Proprietary or commercial disclosure may be found in the Footnotes and Disclosures at the end of this article.
SN - 2666-9145
ER -