Ophthalmic AI With Just Ten Examples: Comparing Multimodal Large Language Models, Vision Foundation Models, and Vision-Language Models
- 1John F. Hardesty M.D. Department of Ophthalmology and Visual Sciences, Washington University in St Louis School of Medicine
Abstract
Purpose: To compare multimodal large language models (MLLMs), vision foundation models (VFMs), and contrastive vision-language models (VLMs) for ophthalmic image interpretation with no labeled examples (zero-shot) or as few as 10 labeled examples per class (10-shot). Design: Retrospective benchmarking study. Subjects: 4 public datasets (UCSD, IDRiD, REFUGE, ASOCT-MSDC) and 3 private datasets (OHTS, BPEI-DLS, WashU-OCT) of anterior- and posterior-segment ophthalmic imaging. Methods: Nine AI models were evaluated across 3 architectural families: MLLMs (Qwen3.8-27B, GLM-5.3-Flash, Gemma-4-31B-it), VFMs (DINOv3, RETFound, VisionFM), and VLMs (CLIP, EyeCLIP, SigLIP2). Models were assessed on disease classification, disease risk and severity grading, anatomical assessment, and lesion localization in fundus and OCT imaging. MLLMs were also compared with a 3D VFM (OCTCube-M) for whole-volume OCT classification. MLLMs were evaluated zero-shot and 10-shot, without additional training. Vision models (VFMs and VLMs) were adapted by linear probing and fine-tuning on the same 10-shot set and on the full training dataset (FD). Main Outcome Measures: Disease classification performance (macro-averaged F1, AUROC, quadratic weighted kappa [QWK]), anatomical localization accuracy, and cross-dataset generalization. Results: For retinal OCT disease classification, 10-shot MLLMs achieved a macro-F1 of 0.94, versus 0.59 (10-shot) and 0.96 (FD) for vision models. For diabetic retinopathy and macular edema grading, 10-shot MLLMs achieved QWKs of 0.69 and 0.73, versus 0.48 and 0.57 (10-shot) and 0.72 and 0.79 (FD) for vision models. For lesion localization, 10-shot MLLMs achieved an F1 of 0.75, versus 0.70 (10-shot) and 0.86 (FD) for vision models. For vertical cup-to-disc ratio estimation (OHTS), zero-shot MLLMs achieved a mean absolute error of 0.14, versus 0.21 (10-shot) for vision models. For whole-volume OCT classification (WashU-OCT), zero-shot MLLMs achieved a macro-F1 of 0.71, versus 0.50 (10-shot) and 0.82 (FD) for vision models. Conclusions: Open-weight MLLMs given 10 labeled examples per class outperformed VFMs and VLMs trained on the same examples in most tasks and approached the performance of fully trained VFMs and VLMs in selected disease classification and grading tasks. MLLMs also demonstrated promising zero-shot performance on non-public clinical datasets without task-specific training. By reducing the need for large labeled datasets and task-specific model adaptation, MLLMs could broaden who can develop and validate ophthalmic AI.
Related
- HalluciGuard-eye: knowledge graph-enhanced retrieval-augmented generation for hallucination detection and mitigation in ophthalmic multimodal large language models
- Artificial intelligence for ophthalmic vessel segmentation: from multimodal imaging to clinical translation
- Feasibility and Diagnostic Potential of Multimodal Neuro-Ophthalmic Biomarkers in Young-Onset Dementia and Functional Cognitive Disorder.
- Extracting smoking history from clinical notes for lung cancer screening decision support: comparing a structured-judgment model with general-purpose large language models
- Image-Based Breast Implant Detection for Mammography Dataset Curation and Near-Real-Time Deployment: Comparing Foundation Models and Task-Specific Convolutional Models
BibTeX
@misc{ramakrishna2026ophthalmic,
title = {{Ophthalmic AI With Just Ten Examples: Comparing Multimodal Large Language Models, Vision Foundation Models, and Vision-Language Models}},
author = {P. Ramakrishna and J. Strand and S. Song and M. B. Tavakoli and S. Zetumer and S. Majidi and Y. Jiang and M. Shen and A. Berni and G. Gregori and P. J. Rosenfeld and M. Kass and M. Gordon and C. S. Lee and Y. Kihara and A. Y. Lee},
howpublished = {medRxiv},
year = {2026},
month = oct,
doi = {10.64898/2026.10.30.26366929},
url = {https://www.medrxiv.org/content/10.64898/2026.10.30.26366929v1},
note = {CC BY-NC-ND},
}
RIS
TY - UNPB
TI - Ophthalmic AI With Just Ten Examples: Comparing Multimodal Large Language Models, Vision Foundation Models, and Vision-Language Models
AU - P. Ramakrishna
AU - J. Strand
AU - S. Song
AU - M. B. Tavakoli
AU - S. Zetumer
AU - S. Majidi
AU - Y. Jiang
AU - M. Shen
AU - A. Berni
AU - G. Gregori
AU - P. J. Rosenfeld
AU - M. Kass
AU - M. Gordon
AU - C. S. Lee
AU - Y. Kihara
AU - A. Y. Lee
PY - 2026
DA - 2026/10/30
PB - medRxiv
DO - 10.64898/2026.10.30.26366929
UR - https://www.medrxiv.org/content/10.64898/2026.10.30.26366929v1
AB - Purpose: To compare multimodal large language models (MLLMs), vision foundation models (VFMs), and contrastive vision-language models (VLMs) for ophthalmic image interpretation with no labeled examples (zero-shot) or as few as 10 labeled examples per class (10-shot). Design: Retrospective benchmarking study. Subjects: 4 public datasets (UCSD, IDRiD, REFUGE, ASOCT-MSDC) and 3 private datasets (OHTS, BPEI-DLS, WashU-OCT) of anterior- and posterior-segment ophthalmic imaging. Methods: Nine AI models were evaluated across 3 architectural families: MLLMs (Qwen3.8-27B, GLM-5.3-Flash, Gemma-4-31B-it), VFMs (DINOv3, RETFound, VisionFM), and VLMs (CLIP, EyeCLIP, SigLIP2). Models were assessed on disease classification, disease risk and severity grading, anatomical assessment, and lesion localization in fundus and OCT imaging. MLLMs were also compared with a 3D VFM (OCTCube-M) for whole-volume OCT classification. MLLMs were evaluated zero-shot and 10-shot, without additional training. Vision models (VFMs and VLMs) were adapted by linear probing and fine-tuning on the same 10-shot set and on the full training dataset (FD). Main Outcome Measures: Disease classification performance (macro-averaged F1, AUROC, quadratic weighted kappa [QWK]), anatomical localization accuracy, and cross-dataset generalization. Results: For retinal OCT disease classification, 10-shot MLLMs achieved a macro-F1 of 0.94, versus 0.59 (10-shot) and 0.96 (FD) for vision models. For diabetic retinopathy and macular edema grading, 10-shot MLLMs achieved QWKs of 0.69 and 0.73, versus 0.48 and 0.57 (10-shot) and 0.72 and 0.79 (FD) for vision models. For lesion localization, 10-shot MLLMs achieved an F1 of 0.75, versus 0.70 (10-shot) and 0.86 (FD) for vision models. For vertical cup-to-disc ratio estimation (OHTS), zero-shot MLLMs achieved a mean absolute error of 0.14, versus 0.21 (10-shot) for vision models. For whole-volume OCT classification (WashU-OCT), zero-shot MLLMs achieved a macro-F1 of 0.71, versus 0.50 (10-shot) and 0.82 (FD) for vision models. Conclusions: Open-weight MLLMs given 10 labeled examples per class outperformed VFMs and VLMs trained on the same examples in most tasks and approached the performance of fully trained VFMs and VLMs in selected disease classification and grading tasks. MLLMs also demonstrated promising zero-shot performance on non-public clinical datasets without task-specific training. By reducing the need for large labeled datasets and task-specific model adaptation, MLLMs could broaden who can develop and validate ophthalmic AI.
ER -