DINOspec: Efficient Multimodal Alignment of Vision and Spectral Foundation Models for Astronomy
Abstract
Astronomical observations provide multimodal views of physical systems, with images and spectra capturing complementary properties of celestial objects. Scientific foundation models can learn powerful representations from these observations, but representations learned by separate models remain difficult to combine. We investigate whether laten space representations learned by separate vision and spectral models can be aligned without retraining their encoders. We introduce DINOspec, a multimodal framework that aligns a frozen DINOv3 image encoder with a pre-trained AION-1 spectral encoder using lightweight adapters and contrastive learning on 20,472 paired images and spectra of astronomical objects. DINOspec improves galaxy morphology classification (F1: 0.72→0.78) and spectral type classification (F1: 0.70→0.74) while training at few as 21M parameters. Redshift prediction shows no consistent improvement over the AION-1 baseline of R2≈0.9, demonstrating that potential benefits of alignment depends on the configuration and downstream task. These results provide an empirical study of composing astronomical representations from independently pretrained foundation models through lightweight alignment modules.
Related
- Gestalt: a meta-foundation model for astronomy
- Star Light, Star Bright: Can Multimodal Models Parse an Astronomy Visual Question Answering Dataset Right?
- Efficient Identification and Analysis of Space debris Trails in Time-Domain Astronomy
- MAGIKS: Multimodal Alignment and Gravitational-Wave-Conditioned Interaction for Kilonova Search
- Transitioning from ADS to SciX to Serve 21st Century Astronomy
BibTeX
@misc{lastufka2026dinospec,
title = {{DINOspec: Efficient Multimodal Alignment of Vision and Spectral Foundation Models for Astronomy}},
author = {Erica Lastufka and Mariia Drozdova and Daniel Schaerer and Svyatoslav Voloshynovskiy},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2608.30503},
eprint = {2608.30503},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2608.30503},
note = {CC BY-NC-ND 4.0},
}
RIS
TY - UNPB
TI - DINOspec: Efficient Multimodal Alignment of Vision and Spectral Foundation Models for Astronomy
AU - Erica Lastufka
AU - Mariia Drozdova
AU - Daniel Schaerer
AU - Svyatoslav Voloshynovskiy
PY - 2026
DA - 2026/10/28
PB - arXiv
DO - 10.48550/arxiv.2608.30503
UR - https://arxiv.org/abs/2608.30503
AB - Astronomical observations provide multimodal views of physical systems, with images and spectra capturing complementary properties of celestial objects. Scientific foundation models can learn powerful representations from these observations, but representations learned by separate models remain difficult to combine. We investigate whether laten space representations learned by separate vision and spectral models can be aligned without retraining their encoders. We introduce DINOspec, a multimodal framework that aligns a frozen DINOv3 image encoder with a pre-trained AION-1 spectral encoder using lightweight adapters and contrastive learning on 20,472 paired images and spectra of astronomical objects. DINOspec improves galaxy morphology classification (F1: 0.72→0.78) and spectral type classification (F1: 0.70→0.74) while training at few as 21M parameters. Redshift prediction shows no consistent improvement over the AION-1 baseline of R^2≈0.9, demonstrating that potential benefits of alignment depends on the configuration and downstream task. These results provide an empirical study of composing astronomical representations from independently pretrained foundation models through lightweight alignment modules.
ER -