Learning Radio Astronomical Representations with LeJEPA and Very Small Models
Abstract
Representations learned by vision foundation models pretrained on natural images have been shown to be useful for out-of-domain astronomical images. Performance on scientific downstream tasks increases with model size, which both carries higher inference costs and limits scalability, even when considering parameter-efficient adaptation. An alternative is to learn representations directly from astronomical observations rather than natural images, through self-supervised pretraining. We evaluate LeJEPA's ability to learn robust representations using very small vision models (∼6M parameters) pretrained on Radio Galaxy Zoo images, comparing with established self-supervised frameworks. We test whether LeJEPA's latent-space regularization leads to better radio galaxy morphology classification. Across three evaluation datasets, LeJEPA achieves performance comparable to a substantially larger foundation model while producing more consistent representations across training and evaluation datasets. These results suggest that the choice of representation learning objective is critical for enabling small domain-specific models to achieve performance competitive with representations transferred from large foundation models in scientific imaging.
Related
- radio-astro-tools: linking radio astronomical data to the astronomical Python ecosystem
- 3-Methoxy-1-propanol Rotational Spectroscopy and Radio Astronomical Search toward NGC 6334I
- 3-methoxy-1-propanol Rotational Spectroscopy and Radio Astronomical Search toward NGC 6334I
- SIFARI: Self-Supervised Interferometric Fitting for Astronomical Radio Imaging
- Object-independent evaluation of small-data astronomical image classifiers: architecture, representation, and acquisition shortcuts
BibTeX
@misc{lastufka2026learning,
title = {{Learning Radio Astronomical Representations with LeJEPA and Very Small Models}},
author = {Erica Lastufka and Mariia Drozdova and Vitaliy Kinakh and Taras Holotyak and Miroslava Dessuages-Zavadsky and Daniel Schaerer and Svyatoslav Voloshynovskiy},
howpublished = {arXiv},
year = {2026},
month = aug,
doi = {10.48550/arxiv.2608.30594},
eprint = {2608.30594},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2608.30594},
note = {CC BY-NC-ND 4.0},
}
RIS
TY - UNPB
TI - Learning Radio Astronomical Representations with LeJEPA and Very Small Models
AU - Erica Lastufka
AU - Mariia Drozdova
AU - Vitaliy Kinakh
AU - Taras Holotyak
AU - Miroslava Dessuages-Zavadsky
AU - Daniel Schaerer
AU - Svyatoslav Voloshynovskiy
PY - 2026
DA - 2026/08/31
PB - arXiv
DO - 10.48550/arxiv.2608.30594
UR - https://arxiv.org/abs/2608.30594
AB - Representations learned by vision foundation models pretrained on natural images have been shown to be useful for out-of-domain astronomical images. Performance on scientific downstream tasks increases with model size, which both carries higher inference costs and limits scalability, even when considering parameter-efficient adaptation. An alternative is to learn representations directly from astronomical observations rather than natural images, through self-supervised pretraining. We evaluate LeJEPA's ability to learn robust representations using very small vision models (∼6M parameters) pretrained on Radio Galaxy Zoo images, comparing with established self-supervised frameworks. We test whether LeJEPA's latent-space regularization leads to better radio galaxy morphology classification. Across three evaluation datasets, LeJEPA achieves performance comparable to a substantially larger foundation model while producing more consistent representations across training and evaluation datasets. These results suggest that the choice of representation learning objective is critical for enabling small domain-specific models to achieve performance competitive with representations transferred from large foundation models in scientific imaging.
ER -