Multimodal OCT–OCTA Classification of Dry AMD, Active MNV, and Inactive MNV Using Frozen Vision Transformer Features: A Patient-Level Nested Cross-Validation Study
- 1Department of Ophthalmology, Faculty of Medicine, Turgut Özal Medical Center, İnönü University, Malatya 44280, Türkiye
- 2Department of Ophthalmology, Faculty of Medicine, Malatya Turgut Özal University, Malatya 44210, Türkiye
- 3Department of Computer Engineering, Faculty of Engineering, Fırat University, Elazig 23119, Türkiye
Abstract
Background/Objectives: Distinguishing dry age-related macular degeneration (AMD) from active and inactive macular neovascularization (MNV) may benefit from integrated interpretation of optical coherence tomography (OCT) and OCT angiography (OCTA). However, the added value of OCTA under artifact-prone conditions remains uncertain. We compared OCT-only, direct OCTA fusion, and OCT–OCTA multimodal classification using frozen Vision Transformer features. Methods: This retrospective study included 368 eyes from 200 patients. OCT B-scans and outer retina and choriocapillaris OCTA images were analyzed. A pretrained Vision Transformer B/32 (ViT-B/32) generated modality-specific embeddings. Class-weighted linear support vector machine (SVM) models were developed using patient-grouped repeated nested cross-validation (10 repetitions; five outer and four inner folds). The primary gradable cohort comprised 320 eyes from 185 patients; severe OCTA artifacts and non-exudative MNV were assessed separately in predefined challenge cohorts. Results: Direct multimodal fusion achieved an accuracy of 0.753, balanced accuracy of 0.679, macro-F1 of 0.685, and macro area under the receiver operating characteristic curve (AUROC) of 0.887. It significantly outperformed direct OCTA fusion across five prespecified metrics; differences from OCT alone were not statistically significant. Class-specific sensitivities were 0.916 for dry AMD, 0.709 for active MNV, and 0.412 for inactive MNV. Performance estimates were similar in the real-world cohort (accuracy, 0.758; macro-AUROC, 0.886). Whole-modality occlusion suggested predominant OCT reliance for dry AMD and greater outer retina OCTA reliance for active and inactive MNV. Conclusions: Frozen Vision Transformer-derived multimodal features supported three-class OCT–OCTA classification under patient-level validation. Structural OCT provided most of the discriminative information. OCTA influenced model predictions in occlusion analyses, but its incremental classification benefit was not established. External multicenter validation is required before clinical use.
Related
- MedFuse: dual-stream fusion of convolutional and vision transformer-based features for enhanced medical image classification
- Transformer-Based Multimodal Fusion Integrating Computed Tomography, Pathology, and Clinical Features for Oral Squamous Cell Carcinoma Recurrence Prediction
- Hybrid deep learning and meta learning for Alzheimer disease classification via multimodal radiomic and visual features
- Prospective validation of multimodal risk stratification using clinical and quantitative imaging features for 12-month outcomes after minimally invasive surgery for lumbar disc herniation
- DeepFusion-HSK: a multimodal deep learning for subtype classification of herpes simplex keratitis with multicenter validation
BibTeX
@article{fırat2026multimodal,
title = {{Multimodal OCT–OCTA Classification of Dry AMD, Active MNV, and Inactive MNV Using Frozen Vision Transformer Features: A Patient-Level Nested Cross-Validation Study}},
author = {İlknur Tuncer Fırat and Murat Fırat and Emrah Öztürk and Taner Tuncer},
journal = {Journal of Clinical Medicine},
year = {2026},
month = nov,
doi = {10.3390/jcm15218647},
url = {https://www.mdpi.com/2077-0383/15/21/8647},
note = {CC BY 4.0},
}
RIS
TY - JOUR
TI - Multimodal OCT–OCTA Classification of Dry AMD, Active MNV, and Inactive MNV Using Frozen Vision Transformer Features: A Patient-Level Nested Cross-Validation Study
AU - İlknur Tuncer Fırat
AU - Murat Fırat
AU - Emrah Öztürk
AU - Taner Tuncer
PY - 2026
DA - 2026/11/06
JO - Journal of Clinical Medicine
DO - 10.3390/jcm15218647
UR - https://www.mdpi.com/2077-0383/15/21/8647
AB - Background/Objectives: Distinguishing dry age-related macular degeneration (AMD) from active and inactive macular neovascularization (MNV) may benefit from integrated interpretation of optical coherence tomography (OCT) and OCT angiography (OCTA). However, the added value of OCTA under artifact-prone conditions remains uncertain. We compared OCT-only, direct OCTA fusion, and OCT–OCTA multimodal classification using frozen Vision Transformer features. Methods: This retrospective study included 368 eyes from 200 patients. OCT B-scans and outer retina and choriocapillaris OCTA images were analyzed. A pretrained Vision Transformer B/32 (ViT-B/32) generated modality-specific embeddings. Class-weighted linear support vector machine (SVM) models were developed using patient-grouped repeated nested cross-validation (10 repetitions; five outer and four inner folds). The primary gradable cohort comprised 320 eyes from 185 patients; severe OCTA artifacts and non-exudative MNV were assessed separately in predefined challenge cohorts. Results: Direct multimodal fusion achieved an accuracy of 0.753, balanced accuracy of 0.679, macro-F1 of 0.685, and macro area under the receiver operating characteristic curve (AUROC) of 0.887. It significantly outperformed direct OCTA fusion across five prespecified metrics; differences from OCT alone were not statistically significant. Class-specific sensitivities were 0.916 for dry AMD, 0.709 for active MNV, and 0.412 for inactive MNV. Performance estimates were similar in the real-world cohort (accuracy, 0.758; macro-AUROC, 0.886). Whole-modality occlusion suggested predominant OCT reliance for dry AMD and greater outer retina OCTA reliance for active and inactive MNV. Conclusions: Frozen Vision Transformer-derived multimodal features supported three-class OCT–OCTA classification under patient-level validation. Structural OCT provided most of the discriminative information. OCTA influenced model predictions in occlusion analyses, but its incremental classification benefit was not established. External multicenter validation is required before clinical use.
SN - 2077-0383
ER -