The Great Chat in the Sky: Aligning Exotic Astronomical Data to Language with AstroBridge
Abstract
Astronomical observations include images, spectra, and light curves, while the scientific literature describes these objects in natural language. We present AstroBridge, a framework that connects these observations to an LLM through a literature-assisted caption generation pipeline. We used catalog metadata to match observations with scientific texts that describe the same objects. A vision-language model generates descriptive captions from representations of observations, matched passages, and available metadata, creating observation-caption pairs without manual captioning. We train AstroBridge to predict captions directly from raw observations. Modality-specific frozen encoders extract representations that a shared, trainable fusion network aligns with the embedding space of an LLM. In caption-based evaluation of spectra, AstroBridge achieves 91.5% accuracy in distinguishing quasars from galaxies, compared to 69.0% from the base model prior to fine-tuning. Limitations remain in galaxy morphology, classification of minority supernova classes, and the factual quality of generated captions. We believe that AstroBridge's framework is a step towards the goal of aligning the totality of astronomy's diverse, exotic data through the lingua franca of text.
Related
- AstroSpecLM: A Spectrum-Language Model for Evidence-Grounded Astronomical Spectral Analysis
- Automated, auditable dating of the Babylonian Astronomical Diaries with language models and astronomical tools
- Executable verification through formalized expert reasoning in astronomical spectroscopy
- The Astronomical Hub: A Unified Ecosystem for Modern Astronomical Research
- radio-astro-tools: linking radio astronomical data to the astronomical Python ecosystem
BibTeX
@misc{habib2026great,
title = {{The Great Chat in the Sky: Aligning Exotic Astronomical Data to Language with AstroBridge}},
author = {Ibrahim Habib and Kevin Li and Trung Nguyen and Gabriel A. Patron and Gaurav Sekar and Nolan Koblischke and Matthieu Le Lain and Shashwat Sourav and Aman Kumar and Kartheik G. Iyer and Joshua S. Speagle and Itai Sfaradi and Michael J. Smith and Ioana Ciucă},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.36375},
eprint = {2610.36375},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.36375},
}
RIS
TY - UNPB
TI - The Great Chat in the Sky: Aligning Exotic Astronomical Data to Language with AstroBridge
AU - Ibrahim Habib
AU - Kevin Li
AU - Trung Nguyen
AU - Gabriel A. Patron
AU - Gaurav Sekar
AU - Nolan Koblischke
AU - Matthieu Le Lain
AU - Shashwat Sourav
AU - Aman Kumar
AU - Kartheik G. Iyer
AU - Joshua S. Speagle
AU - Itai Sfaradi
AU - Michael J. Smith
AU - Ioana Ciucă
PY - 2026
DA - 2026/10/26
PB - arXiv
DO - 10.48550/arxiv.2610.36375
UR - https://arxiv.org/abs/2610.36375
AB - Astronomical observations include images, spectra, and light curves, while the scientific literature describes these objects in natural language. We present AstroBridge, a framework that connects these observations to an LLM through a literature-assisted caption generation pipeline. We used catalog metadata to match observations with scientific texts that describe the same objects. A vision-language model generates descriptive captions from representations of observations, matched passages, and available metadata, creating observation-caption pairs without manual captioning. We train AstroBridge to predict captions directly from raw observations. Modality-specific frozen encoders extract representations that a shared, trainable fusion network aligns with the embedding space of an LLM. In caption-based evaluation of spectra, AstroBridge achieves 91.5% accuracy in distinguishing quasars from galaxies, compared to 69.0% from the base model prior to fine-tuning. Limitations remain in galaxy morphology, classification of minority supernova classes, and the factual quality of generated captions. We believe that AstroBridge's framework is a step towards the goal of aligning the totality of astronomy's diverse, exotic data through the lingua franca of text.
ER -