Benchmarking Open-Source Vision-Language Models for Brain Metastasis Assessment on Single-Slice Contrast-Enhanced MRI
- 1Department of Radiology, Seoul St. Mary's Hospital, College of Medicine, The Catholic University of Korea
Abstract
PurposeOpen-source vision-language models (VLMs) can be locally deployed without external internet access. This study aimed to evaluate the slice-level lesion-detection performance of multiple general-purpose and medical-purpose open-source VLMs for brain metastases on contrast-enhanced (CE) MRI, and to assess their accuracy in characterizing detailed imaging features on lesion-positive images.
Materials and MethodsOne hundred lesion-positive axial CE T1-weighted images and 100 matched lesion-negative images from 100 patients were analyzed using eight evaluable VLMs: five general-purpose VLMs-InternVL3-8B, Qwen2.5-VL-7B-Instruct, MiniCPM-V-4.5, DeepSeek-VL2-tiny, and Phi-3.5-vision-instruct-and three medical-purpose VLMs-MedGemma-4B-it, LLaVA-Med v1.5, and HuatuoGPT-Vision-7B. Lesion-detection performance was assessed using sensitivity, specificity, and balanced accuracy. On lesion-positive images, accuracy was evaluated for lesion count, laterality, anatomic location, enhancement pattern, necrosis, and mass effect. Balanced accuracy was reported with patient-level cluster bootstrap 95% confidence intervals (CIs). Models were compared using the Friedman test and paired patient-level permutation tests with Benjamini-Hochberg correction. Secondary outcomes were compared with majority-class baselines using exact McNemar tests. A revised prompt was tested post hoc.
ResultsThe median age of the study patients was 65 years (IQR, 59.0-70.0), and 58 patients were male (58.0%). MiniCPM-V-4.5 showed significantly higher balanced accuracy (80.5%; 95% CI, 75.0-85.5) than all other models (FDR-adjusted P < 0.001). Seven of the eight models showed predicted-positive rates of 81-100%, and 70.8% (17/24) of responses to synthetic control images without anatomic content were classified as lesion-positive. With the revised prompt, balanced accuracy increased from 59.5% to 77.0% in HuatuoGPT-Vision-7B and from 51.5% to 68.5% in InternVL3-8B. No model significantly outperformed the majority-class baseline for enhancement pattern or for mass effect.
ConclusionThis slice-level technical benchmark demonstrated variable lesion-detection performance of open-source VLMs for brain metastasis in an image-only setting, with most models showing a strong positive-response tendency.
Related
- Predicting high-burden central compartment lymph node metastasis (≥ 3 positive nodes) in papillary thyroid carcinoma: a habitat radiomic framework based on reconstructed contrast-enhanced CT
- Contrast-enhanced ultrasound in assessment of testicular injury based on different degrees of testicular torsion in rabbits
- Multimodal radiomics based on dynamic contrast-enhanced ultrasound for predicting cervical lymph node metastasis in papillary thyroid carcinoma with Hashimoto’s thyroiditis
- MR radiomics for predicting lymph node metastasis of pancreatic ductal adenocarcinoma by comparing different contrast-enhanced phases
- Assessment of pleural effusion using intracavitary contrast-enhanced ultrasound: a pilot study
BibTeX
@misc{kim2026benchmarking,
title = {{Benchmarking Open-Source Vision-Language Models for Brain Metastasis Assessment on Single-Slice Contrast-Enhanced MRI}},
author = {J. Kim and B.-s. Kim and J. S. Ko and J. Dong and S. Y. Youn and J. Jang and K.-J. Ahn},
howpublished = {medRxiv},
year = {2026},
month = aug,
doi = {10.64898/2026.08.24.26361169},
url = {https://www.medrxiv.org/content/10.64898/2026.08.24.26361169v2},
note = {CC BY-NC-ND},
}
RIS
TY - UNPB
TI - Benchmarking Open-Source Vision-Language Models for Brain Metastasis Assessment on Single-Slice Contrast-Enhanced MRI
AU - J. Kim
AU - B.-s. Kim
AU - J. S. Ko
AU - J. Dong
AU - S. Y. Youn
AU - J. Jang
AU - K.-J. Ahn
PY - 2026
DA - 2026/08/24
PB - medRxiv
DO - 10.64898/2026.08.24.26361169
UR - https://www.medrxiv.org/content/10.64898/2026.08.24.26361169v2
AB - PurposeOpen-source vision-language models (VLMs) can be locally deployed without external internet access. This study aimed to evaluate the slice-level lesion-detection performance of multiple general-purpose and medical-purpose open-source VLMs for brain metastases on contrast-enhanced (CE) MRI, and to assess their accuracy in characterizing detailed imaging features on lesion-positive images. Materials and MethodsOne hundred lesion-positive axial CE T1-weighted images and 100 matched lesion-negative images from 100 patients were analyzed using eight evaluable VLMs: five general-purpose VLMs-InternVL3-8B, Qwen2.5-VL-7B-Instruct, MiniCPM-V-4.5, DeepSeek-VL2-tiny, and Phi-3.5-vision-instruct-and three medical-purpose VLMs-MedGemma-4B-it, LLaVA-Med v1.5, and HuatuoGPT-Vision-7B. Lesion-detection performance was assessed using sensitivity, specificity, and balanced accuracy. On lesion-positive images, accuracy was evaluated for lesion count, laterality, anatomic location, enhancement pattern, necrosis, and mass effect. Balanced accuracy was reported with patient-level cluster bootstrap 95% confidence intervals (CIs). Models were compared using the Friedman test and paired patient-level permutation tests with Benjamini-Hochberg correction. Secondary outcomes were compared with majority-class baselines using exact McNemar tests. A revised prompt was tested post hoc. ResultsThe median age of the study patients was 65 years (IQR, 59.0-70.0), and 58 patients were male (58.0%). MiniCPM-V-4.5 showed significantly higher balanced accuracy (80.5%; 95% CI, 75.0-85.5) than all other models (FDR-adjusted P < 0.001). Seven of the eight models showed predicted-positive rates of 81-100%, and 70.8% (17/24) of responses to synthetic control images without anatomic content were classified as lesion-positive. With the revised prompt, balanced accuracy increased from 59.5% to 77.0% in HuatuoGPT-Vision-7B and from 51.5% to 68.5% in InternVL3-8B. No model significantly outperformed the majority-class baseline for enhancement pattern or for mass effect. ConclusionThis slice-level technical benchmark demonstrated variable lesion-detection performance of open-source VLMs for brain metastasis in an image-only setting, with most models showing a strong positive-response tendency.
ER -