Object-independent evaluation of small-data astronomical image classifiers: architecture, representation, and acquisition shortcuts
Abstract
Reported performance in small astronomical image datasets can reflect object reuse, field-of-view differences, acquisition settings, and catalogue-derived properties as well as astronomical morphology. We designed \dataset{} to measure these effects in a four-class task comprising galaxies, globular clusters, nebulae, and open clusters. The dataset contains 370 catalogue-supported physical objects, each represented by three Digitised Sky Survey FITS bands, and uses a group-constrained split of 222 training, 75 validation, and 73 test objects. We compared a custom CNN--Transformer trained from scratch with ImageNet-pretrained ResNet-50 and ViT-B/16 under an adaptive full-field representation, a common angular field, and two central-region-masked controls, using three fixed seeds. In the one-pass object-independent test, mean balanced accuracy on the adaptive view was 0.773±0.014, 0.874±0.054, and 0.866±0.023 for the custom model, ResNet-50, and ViT-B/16, respectively. No primary architecture contrast rejected its null after Holm correction; ResNet-50 and ViT-B/16 differed by 0.007 balanced-accuracy points (95\% cluster-bootstrap interval $[-0.048,0.062]$). Controls based on acquisition and catalogue metadata reached 0.721--0.735, and central-region-masked adaptive views reached 0.731--0.800. These results demonstrate generalisation to unseen physical objects, but also show that image context, acquisition settings, and catalogue-derived object properties carry substantial class information. In this regime, high accuracy cannot be interpreted as evidence that a classifier learned target morphology, and the comparison does not isolate architecture from pretraining.
Related
- Contrastive Astromer: Two-Stage Contrastive Representation Learning for Astronomical Light Curve Classification
- FAIR-Compliant Architecture for Heterogeneous Astronomical Data: KazVO Framework
- Executable verification through formalized expert reasoning in astronomical spectroscopy
- The Astronomical Hub: A Unified Ecosystem for Modern Astronomical Research
- radio-astro-tools: linking radio astronomical data to the astronomical Python ecosystem
BibTeX
@misc{marinho2026objectindependent,
title = {{Object-independent evaluation of small-data astronomical image classifiers: architecture, representation, and acquisition shortcuts}},
author = {Eraldo Pereira Marinho},
howpublished = {arXiv},
year = {2026},
month = nov,
doi = {10.48550/arxiv.2611.06484},
eprint = {2611.06484},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2611.06484},
}
RIS
TY - UNPB
TI - Object-independent evaluation of small-data astronomical image classifiers: architecture, representation, and acquisition shortcuts
AU - Eraldo Pereira Marinho
PY - 2026
DA - 2026/11/04
PB - arXiv
DO - 10.48550/arxiv.2611.06484
UR - https://arxiv.org/abs/2611.06484
AB - Reported performance in small astronomical image datasets can reflect object reuse, field-of-view differences, acquisition settings, and catalogue-derived properties as well as astronomical morphology. We designed \dataset{} to measure these effects in a four-class task comprising galaxies, globular clusters, nebulae, and open clusters. The dataset contains 370 catalogue-supported physical objects, each represented by three Digitised Sky Survey FITS bands, and uses a group-constrained split of 222 training, 75 validation, and 73 test objects. We compared a custom CNN--Transformer trained from scratch with ImageNet-pretrained ResNet-50 and ViT-B/16 under an adaptive full-field representation, a common angular field, and two central-region-masked controls, using three fixed seeds. In the one-pass object-independent test, mean balanced accuracy on the adaptive view was 0.773±0.014, 0.874±0.054, and 0.866±0.023 for the custom model, ResNet-50, and ViT-B/16, respectively. No primary architecture contrast rejected its null after Holm correction; ResNet-50 and ViT-B/16 differed by 0.007 balanced-accuracy points (95\% cluster-bootstrap interval $[-0.048,0.062]$). Controls based on acquisition and catalogue metadata reached 0.721--0.735, and central-region-masked adaptive views reached 0.731--0.800. These results demonstrate generalisation to unseen physical objects, but also show that image context, acquisition settings, and catalogue-derived object properties carry substantial class information. In this regime, high accuracy cannot be interpreted as evidence that a classifier learned target morphology, and the comparison does not isolate architecture from pretraining.
ER -