Rethinking Domain Specialization for Open-Ended Scientific Reasoning in Astronomy Language Models
Abstract
Domain-specialized language models are widely used for scientific question answering, but stronger general-purpose systems raise a sharper question: when does domain-specific fine-tuning remain valuable for open-ended scientific reasoning? We study this in astronomy with a curated QA benchmark from publicly available 2017--2026 Olympiad-style materials. The free-response subset contains 300 questions, including 204 text-only and 96 image-linked examples. We compare open-weight and API-served general-purpose, multimodal, and astronomy-specialized models using judge-based correctness and complementary reference metrics. Strong general-purpose models establish the highest correctness baseline in this testbed, while analyses of metric agreement, judge sensitivity, benchmark composition, and modality reveal variation not captured by a single leaderboard. These results motivate treating domain specialization as a task- and deployment-dependent property and highlight the role of domain-specific evaluation in determining which models, capabilities, and evaluation criteria are appropriate for scientific workflows.
Related
- Reconstructing Implicit Scientific Knowledge: Evaluating LLM Agents through End-to-End Reproduction of Astronomy
- More than half of recent astronomy papers are written with language-model assistance
- Transitioning from ADS to SciX to Serve 21st Century Astronomy
- From Axion Detection to Dark Matter Astronomy
- Neutrino Astronomy at High Energies - An Experimental Review
BibTeX
@misc{lama2026rethinking,
title = {{Rethinking Domain Specialization for Open-Ended Scientific Reasoning in Astronomy Language Models}},
author = {Vanessa Lama and Sanjay Das and Emily Herron and Yuan-Sen Ting and Tijmen de Haan and Junqi Yin and Tirthankar Ghosal and Feiyi Wang},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2609.17644},
eprint = {2609.17644},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2609.17644},
}
RIS
TY - UNPB
TI - Rethinking Domain Specialization for Open-Ended Scientific Reasoning in Astronomy Language Models
AU - Vanessa Lama
AU - Sanjay Das
AU - Emily Herron
AU - Yuan-Sen Ting
AU - Tijmen de Haan
AU - Junqi Yin
AU - Tirthankar Ghosal
AU - Feiyi Wang
PY - 2026
DA - 2026/09/15
PB - arXiv
DO - 10.48550/arxiv.2609.17644
UR - https://arxiv.org/abs/2609.17644
AB - Domain-specialized language models are widely used for scientific question answering, but stronger general-purpose systems raise a sharper question: when does domain-specific fine-tuning remain valuable for open-ended scientific reasoning? We study this in astronomy with a curated QA benchmark from publicly available 2017--2026 Olympiad-style materials. The free-response subset contains 300 questions, including 204 text-only and 96 image-linked examples. We compare open-weight and API-served general-purpose, multimodal, and astronomy-specialized models using judge-based correctness and complementary reference metrics. Strong general-purpose models establish the highest correctness baseline in this testbed, while analyses of metric agreement, judge sensitivity, benchmark composition, and modality reveal variation not captured by a single leaderboard. These results motivate treating domain specialization as a task- and deployment-dependent property and highlight the role of domain-specific evaluation in determining which models, capabilities, and evaluation criteria are appropriate for scientific workflows.
ER -