FacilityAudit-Agent: Beyond Mention-as-Usage for Fine-Grained Facility Contribution Attribution in Astrophysics Publications
Abstract
Observatory bibliographies increasingly need to determine not only whether a facility is named in a paper, but how its data support the scientific argument. We frame this need as semantic contribution attribution on a six-level scale (L0-L5), from non-use and mention-only cases to auxiliary, major, and exclusive data support. We present FacilityAudit-Agent, a framework that builds a page-bounded outline of each PDF, enriches sections with roles and facility mentions, and uses a tool-calling large language model (LLM) to fetch text on demand. Evidence from the sections is combined through an explicit decision tree and a coverage-calibrated confidence score, with uncertain cases routed to human review. The framework supports both single-facility and inventory audits and transfers to new facilities without retraining. On a test set of 39,358 human-annotated paper-facility pairs, FacilityAudit-Agent achieves 89.1% accuracy and 79.8% macro-F1 in single_audit mode, outperforming the other three baselines by 5.8, 8.3, and 12.1 accuracy points, respectively, with the clearest gains at L2 and L5. In inventory_audit mode, it also leads in facility listing and in level assignment among listed pairs. Ablations show multi-hop reading and tree indexing matter most for L3-L5. FacilityAudit-Agent has been deployed at the National Astronomical Data Center, where it is responsible for processing publications related to LAMOST, FAST, and EP.
Related
- The contribution of Soviet scientists to the emergence of high-energy neutrino astrophysics: from the Cherenkov effect to underwater and in-ice telescopes
- AI's Capability in Assisting Scientific Research in Physics, Astrophysics, and Cosmology I: Literature Review
- AI's Capability in Assisting Scientific Research in Physics, Astrophysics, and Cosmology II: Project Planning and Proposal Evaluation
- Nonlinear Physics and Self-Organized Criticality in Astrophysics
- Promoting Green Computing in High Energy Physics and Astrophysics
BibTeX
@misc{shao2026facilityauditagent,
title = {{FacilityAudit-Agent: Beyond Mention-as-Usage for Fine-Grained Facility Contribution Attribution in Astrophysics Publications}},
author = {Wujun Shao and Dongwei Fan and Daoqi Peng and Yunfei Xu and Chenzhou Cui and Xin Lyu and Wei Huang},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.32135},
eprint = {2610.32135},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.32135},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - FacilityAudit-Agent: Beyond Mention-as-Usage for Fine-Grained Facility Contribution Attribution in Astrophysics Publications
AU - Wujun Shao
AU - Dongwei Fan
AU - Daoqi Peng
AU - Yunfei Xu
AU - Chenzhou Cui
AU - Xin Lyu
AU - Wei Huang
PY - 2026
DA - 2026/10/23
PB - arXiv
DO - 10.48550/arxiv.2610.32135
UR - https://arxiv.org/abs/2610.32135
AB - Observatory bibliographies increasingly need to determine not only whether a facility is named in a paper, but how its data support the scientific argument. We frame this need as semantic contribution attribution on a six-level scale (L0-L5), from non-use and mention-only cases to auxiliary, major, and exclusive data support. We present FacilityAudit-Agent, a framework that builds a page-bounded outline of each PDF, enriches sections with roles and facility mentions, and uses a tool-calling large language model (LLM) to fetch text on demand. Evidence from the sections is combined through an explicit decision tree and a coverage-calibrated confidence score, with uncertain cases routed to human review. The framework supports both single-facility and inventory audits and transfers to new facilities without retraining. On a test set of 39,358 human-annotated paper-facility pairs, FacilityAudit-Agent achieves 89.1% accuracy and 79.8% macro-F1 in single_audit mode, outperforming the other three baselines by 5.8, 8.3, and 12.1 accuracy points, respectively, with the clearest gains at L2 and L5. In inventory_audit mode, it also leads in facility listing and in level assignment among listed pairs. Ablations show multi-hop reading and tree indexing matter most for L3-L5. FacilityAudit-Agent has been deployed at the National Astronomical Data Center, where it is responsible for processing publications related to LAMOST, FAST, and EP.
ER -