{
  "abstract": "Purpose: To develop and validate a rubric-based large language model (LLM) evaluator for assessing the diagnostic quality of LLM-generated age-related macular degeneration (AMD) diagnoses from OCT scans, benchmarked against human evaluators, and, as a secondary aim, to characterize the diagnostic capability of the underlying LLMs.\n\nDesign: Retrospective study primarily developing and validating a rubric-based LLM evaluator for assessing the diagnostic quality of LLM-generated diagnoses on retinal OCT, with the diagnostic performance of the LLMs assessed as a secondary analysis.\n\nParticipants: A total of 140 OCT scans (70 of choroidal neovascular membrane, 70 of drusen), 6 diagnoser models (GPT-4.1, GPT-4o, o4-mini, Gemini-2.5-Flash, GPT-5-mini, and GPT-5.5), and 4 evaluator models (GPT-4.1, GPT-5-mini, o4-mini, and Gemini-2.5-Flash) were included in this study.\n\nMethods: We developed a 2-stage workflow to automate OCT diagnosis associated with AMD. First, LLM “diagnosers” are prompted to provide a diagnosis in natural language from OCT scans. Second, LLM “evaluators” grade the diagnosis against a predetermined rubric for accuracy, imaging features, differential diagnosis, and clarity. A pilot experiment on 20 OCT scans benchmarked four LLM evaluators against a human evaluator using the identical rubric. Interrater reliability between the selected LLM evaluator and 2 retinal specialists was further assessed across 560 image-diagnosis pairs using intraclass correlation coefficients (ICCs) and two-1-sided test equivalence analysis. The best-performing LLM evaluator was then deployed in a full-scale analysis on 140 OCT scans to grade all 6 diagnoser models.\n\nMain Outcome Measures: Average gradings, standard deviation, P value, correlation coefficient, ICC values, paired t-test P values, and two-1-sided test equivalence bounds between human and LLM evaluators.\n\nResults: In the pilot analysis, Gemini-2.5-Flash demonstrated the strongest alignment with the human evaluator. In the expanded experiment across 560 image-diagnosis pairs, ICC analysis confirmed that Gemini-2.5-Flash's agreement with human specialists (ICC = 0.897) was comparable to interspecialist agreement (ICC = 0.806), and two-1-sided test analysis supported statistical equivalence with human scoring, with a mean difference of 0.123 points. In the large-scale evaluation, GPT-5.5 achieved the highest average diagnoser score (6.3/8), followed by GPT-5-mini (5.0) and Gemini-2.5-Flash (4.9).\n\nConclusions: Large language model evaluators demonstrate a high level of agreement with human evaluator in supervising the quality of AMD diagnoses on OCT scans generated by LLM diagnosers. This workflow demonstrates the potential of LLM-powered automation for diagnostic quality evaluation in ophthalmology clinical practice.\n\nFinancial Disclosure(s): Proprietary or commercial disclosure may be found in the Footnotes and Disclosures at the end of this article.",
  "abstract_shown": true,
  "affiliations": [
    "Department of Biomedical Engineering, Duke University, Durham, North Carolina",
    "Department of Electrical and Computer Engineering, Duke University, Durham, North Carolina",
    "Duke Eye Center, Durham, North Carolina",
    "Department of Computer Science, Duke University, Durham, North Carolina",
    "Department of Ophthalmology, Duke University, Durham, North Carolina"
  ],
  "article_type": "research article",
  "arxiv": null,
  "author_total": 7,
  "authors": [
    {
      "affs": [
        0
      ],
      "name": "Ziqi Zhou"
    },
    {
      "affs": [
        1
      ],
      "name": "David Hunt"
    },
    {
      "affs": [
        2
      ],
      "name": "James M. Harris"
    },
    {
      "affs": [
        2
      ],
      "name": "Michael J. Fliotsos"
    },
    {
      "affs": [
        1
      ],
      "name": "Visrut Sudhakar"
    },
    {
      "affs": [
        1,
        3
      ],
      "name": "Miroslav Pajic"
    },
    {
      "affs": [
        1,
        4
      ],
      "name": "Majda Hadziahmetovic"
    }
  ],
  "bibtex": "https://research.draconitic.com/paper/bjnk96sk0hf6d.bib",
  "cited_by": 0,
  "doi": "10.1016/j.xops.2026.101372",
  "fulltext_url": "https://pmc.ncbi.nlm.nih.gov/articles/PMC13696338/",
  "id": "bjnk96sk0hf6d",
  "issns": [
    "2666-9145"
  ],
  "landing_url": "https://doi.org/10.1016/j.xops.2026.101372",
  "language": "en",
  "licence": "CC-BY-NC-ND-4.0",
  "licence_label": "CC BY-NC-ND 4.0",
  "licence_source": "article",
  "licence_url": "https://creativecommons.org/licenses/by-nc-nd/4.0/",
  "metadata_licence": null,
  "native_subjects": [
    [
      "heading",
      "Original Article"
    ],
    [
      "openalex",
      "2741|Retinal Imaging and Analysis"
    ]
  ],
  "openalex": "W7204242071",
  "pdf_url": "https://europepmc.org/articles/PMC13696338?pdf=render",
  "pmcid": "PMC13696338",
  "pmid": "42935864",
  "published": "2026-08-26",
  "publisher": "Elsevier",
  "ris": "https://research.draconitic.com/paper/bjnk96sk0hf6d.ris",
  "sources": [
    {
      "first_seen": "2026-10-30T12:17:42Z",
      "native_id": "PMC13696338",
      "source": "pmc",
      "url": "https://doi.org/10.1016/j.xops.2026.101372"
    },
    {
      "first_seen": "2026-10-30T12:22:24Z",
      "native_id": "MED:42935864",
      "source": "europepmc",
      "url": "https://europepmc.org/article/MED/42935864"
    }
  ],
  "subjects": [
    {
      "field": "medicine",
      "origin": "openalex:2741|Retinal Imaging and Analysis",
      "subfield": "imaging"
    }
  ],
  "title": "Comparative Large Language Models Evaluation for Age-Related Macular Degeneration Diagnosis on OCT and Rubric-Based Diagnostic Quality Assessment",
  "title_text": "Comparative Large Language Models Evaluation for Age-Related Macular Degeneration Diagnosis on OCT and Rubric-Based Diagnostic Quality Assessment",
  "type": "peer-reviewed",
  "updated_at": "2026-10-30T12:33:41Z",
  "url": "https://research.draconitic.com/paper/bjnk96sk0hf6d",
  "venue": "Ophthalmology Science",
  "versions": []
}