Vision-Language Model Ensembles Achieve Human-Expert Accuracy for Galaxy Merger Classification
Abstract
Models (VLMs) combined using a Bayesian statistical framework can classify galaxy merger morphologies with accuracy comparable to trained human experts. We deploy 15 VLM classifier configurations, spanning four model architectures (Gemma-4 E2B, Gemma-4 E4B, Qwen2.5-VL, and Qwen3-VL) tested with up to four prompt engineering strategies each. We evaluate their performance against a truth-known sample of 41 VELA+SUNRISE mock galaxy images from Lambrides et al. (2021a). As a proof-of-concept, all validation is performed on these mock images; application to real observational samples will require additional calibration and observational validation. The VLM ensemble achieves 83.3% accuracy on confident classifications (merger probability pM >= 0.8 or pM <= 0.2) and 58.3% completeness, with 5 misclassified galaxies, compared to 85% for both human accuracy and completeness. The ensemble recovers the population merger fraction to within 0.66 sigma of the truth (fM = 0.52 +/- 0.09 vs. true value of 0.585). Bayesian weighting improves overall accuracy by 17.1 percentage points over simple majority voting, with sensitivity improving by 29.2 percentage points. The ensemble produces 5 misclassifications (2 false positives, 3 false negatives), comparable to the 6 misclassifications (5 false positives, 1 false negative) reported for human classifiers by L21. The error-profile differences are not statistically significant for this sample. VLMs also produce more moderate per-galaxy merger probability distributions (27% uncertain) than the more polarized human distributions (15% uncertain), though this difference is also consistent with statistical fluctuation. These results establish VLMs as scalable, reproducible alternatives to human classifiers within a Bayesian probabilistic merger-fraction framework, for large-survey applications.
Related
- Identifying Scientifically Interesting Galaxies with Vision-Language Models
- Quasi-steady flavor configuration of multi-energy neutrino ensembles
- Differentiable astrophysics at scale: solving and differentiating ODE ensembles on the GPU
- Accelerating massive ensembles of ordinary differential equations
- Reducing False Positives in Strong-Lens Searches with Generalized-Mean Consensus of Machine-Learning Ensembles in the Kilo-Degree Survey
BibTeX
@misc{chiaberge2026visionlanguage,
title = {{Vision-Language Model Ensembles Achieve Human-Expert Accuracy for Galaxy Merger Classification}},
author = {Marco Chiaberge and Elias Stengel-Eskin and Massimo Stiavelli and Colin Norman},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2606.00415},
eprint = {2606.00415},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2606.00415},
}
RIS
TY - UNPB
TI - Vision-Language Model Ensembles Achieve Human-Expert Accuracy for Galaxy Merger Classification
AU - Marco Chiaberge
AU - Elias Stengel-Eskin
AU - Massimo Stiavelli
AU - Colin Norman
PY - 2026
DA - 2026/09/24
PB - arXiv
DO - 10.48550/arxiv.2606.00415
UR - https://arxiv.org/abs/2606.00415
AB - Models (VLMs) combined using a Bayesian statistical framework can classify galaxy merger morphologies with accuracy comparable to trained human experts. We deploy 15 VLM classifier configurations, spanning four model architectures (Gemma-4 E2B, Gemma-4 E4B, Qwen2.5-VL, and Qwen3-VL) tested with up to four prompt engineering strategies each. We evaluate their performance against a truth-known sample of 41 VELA+SUNRISE mock galaxy images from Lambrides et al. (2021a). As a proof-of-concept, all validation is performed on these mock images; application to real observational samples will require additional calibration and observational validation. The VLM ensemble achieves 83.3% accuracy on confident classifications (merger probability pM >= 0.8 or pM <= 0.2) and 58.3% completeness, with 5 misclassified galaxies, compared to 85% for both human accuracy and completeness. The ensemble recovers the population merger fraction to within 0.66 sigma of the truth (fM = 0.52 +/- 0.09 vs. true value of 0.585). Bayesian weighting improves overall accuracy by 17.1 percentage points over simple majority voting, with sensitivity improving by 29.2 percentage points. The ensemble produces 5 misclassifications (2 false positives, 3 false negatives), comparable to the 6 misclassifications (5 false positives, 1 false negative) reported for human classifiers by L21. The error-profile differences are not statistically significant for this sample. VLMs also produce more moderate per-galaxy merger probability distributions (27% uncertain) than the more polarized human distributions (15% uncertain), though this difference is also consistent with statistical fluctuation. These results establish VLMs as scalable, reproducible alternatives to human classifiers within a Bayesian probabilistic merger-fraction framework, for large-survey applications.
ER -