Do Astronomical Foundation Models Know When They Will Fail?
Abstract
Astronomical foundation models are increasingly used as reusable representations across surveys, but it is unclear when the same object remains consistently represented across instruments or whether a model can warn us before transfer fails. We present a conformal framework for auditing cross-survey representations using cross-matched Legacy Survey--HSC, HSC--JWST, and SDSS--HSC objects. Astronomy-specific AstroPT embeddings show lower translation errors and tighter conformal regions than general-purpose visual backbones at comparable coverage, while different model families fail on different objects. We then use adaptive conformal prediction to test whether the source embedding contains information about its own future error. Using a ridge difficulty model fitted to source embeddings, we find that this warning signal is consistent across the tested backbones for Legacy→HSC, backbone-dependent for HSC→JWST, and largely absent for SDSS→HSC. When informative, adaptation improves coverage for the hardest objects. Matched-control analysis further reveals silent failures not explained by magnitude, apparent size, or nearest-neighbor distance. Hence, we show that conformal uncertainty provides a realistic way to identify when astronomical representations transfer reliably and when they fail without warning.
Related
- Do Tabular Foundation Models Know Physics? Contamination, Units, and the Deterministic Limit
- Will a supernova explode in CD-3011223?
- Executable verification through formalized expert reasoning in astronomical spectroscopy
- The Astronomical Hub: A Unified Ecosystem for Modern Astronomical Research
- radio-astro-tools: linking radio astronomical data to the astronomical Python ecosystem
BibTeX
@misc{bulusu2026astronomical,
title = {{Do Astronomical Foundation Models Know When They Will Fail?}},
author = {Vaidehi Bulusu and Mehul Goyal and Md Rabius Sany Apu and Michael J. Smith and Shashwat Sourav},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.02613},
eprint = {2610.02613},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.02613},
note = {CC BY-SA 4.0},
}
RIS
TY - UNPB
TI - Do Astronomical Foundation Models Know When They Will Fail?
AU - Vaidehi Bulusu
AU - Mehul Goyal
AU - Md Rabius Sany Apu
AU - Michael J. Smith
AU - Shashwat Sourav
PY - 2026
DA - 2026/10/02
PB - arXiv
DO - 10.48550/arxiv.2610.02613
UR - https://arxiv.org/abs/2610.02613
AB - Astronomical foundation models are increasingly used as reusable representations across surveys, but it is unclear when the same object remains consistently represented across instruments or whether a model can warn us before transfer fails. We present a conformal framework for auditing cross-survey representations using cross-matched Legacy Survey--HSC, HSC--JWST, and SDSS--HSC objects. Astronomy-specific AstroPT embeddings show lower translation errors and tighter conformal regions than general-purpose visual backbones at comparable coverage, while different model families fail on different objects. We then use adaptive conformal prediction to test whether the source embedding contains information about its own future error. Using a ridge difficulty model fitted to source embeddings, we find that this warning signal is consistent across the tested backbones for Legacy→HSC, backbone-dependent for HSC→JWST, and largely absent for SDSS→HSC. When informative, adaptation improves coverage for the hardest objects. Matched-control analysis further reveals silent failures not explained by magnitude, apparent size, or nearest-neighbor distance. Hence, we show that conformal uncertainty provides a realistic way to identify when astronomical representations transfer reliably and when they fail without warning.
ER -