Publications as benchmarks: auditing AI agents on spatial transcriptomics alignment
Abstract
AI agents are increasingly automating data analysis, but responsible adoption requires reliable performance evaluation, which is costly in specialized scientific domains. We introduce a framework that translates scientific publications into auditable benchmarks for AI agents and demonstrate its use on spatial transcriptomics alignment tasks. Across 121 tasks, 16 baseline methods, and six AI agents, the best agents (GPT-6-Astra and Gemini-4-Argon) matched or outperformed all baselines on 80% of tasks, often using surprisingly simple heuristics. Agent performance was robust to prompting and access to pre-installed packages. Benchmark construction also uncovered errors in prior work and shortcuts in current evaluation metrics. Finally, we show that agent-generated solutions can be distilled into a reusable program that ranks in the top 10% of baseline methods on unseen data. Our framework offers a scalable way to evaluate AI agents in specialized scientific domains by leveraging and stress-testing existing papers and metrics.
Related
- Alignment with sex-typical brain architecture: Links to mood pathology, sex hormones and transcriptomics of neurodegenerative conditions
- PRISM: probabilistic cell-type resolution with integrated spatial modelling for simultaneous ambient RNA decontamination and doublet correction in spatial transcriptomics
- Spatial transcriptomics profiling across tissue boundaries in the developing mouse brain
- SpaCoEx: Sparse Gene Selection for Spatially Varying Co-expression in Spatial Transcriptomics
- NicheCorr: Identifying Niche-Specific Differential Gene Co-Expression in Spatial Transcriptomics
BibTeX
@misc{chen2026publications,
title = {{Publications as benchmarks: auditing AI agents on spatial transcriptomics alignment}},
author = {Yiqun T. Chen and Jan Matthias and Jianing Yao and Hongyu Eric Zhao and Stephanie C. Hicks},
howpublished = {arXiv},
year = {2026},
month = nov,
doi = {10.48550/arxiv.2611.11796},
eprint = {2611.11796},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2611.11796},
note = {CC BY-NC-ND 4.0},
}
RIS
TY - UNPB
TI - Publications as benchmarks: auditing AI agents on spatial transcriptomics alignment
AU - Yiqun T. Chen
AU - Jan Matthias
AU - Jianing Yao
AU - Hongyu Eric Zhao
AU - Stephanie C. Hicks
PY - 2026
DA - 2026/11/09
PB - arXiv
DO - 10.48550/arxiv.2611.11796
UR - https://arxiv.org/abs/2611.11796
AB - AI agents are increasingly automating data analysis, but responsible adoption requires reliable performance evaluation, which is costly in specialized scientific domains. We introduce a framework that translates scientific publications into auditable benchmarks for AI agents and demonstrate its use on spatial transcriptomics alignment tasks. Across 121 tasks, 16 baseline methods, and six AI agents, the best agents (GPT-6-Astra and Gemini-4-Argon) matched or outperformed all baselines on 80% of tasks, often using surprisingly simple heuristics. Agent performance was robust to prompting and access to pre-installed packages. Benchmark construction also uncovered errors in prior work and shortcuts in current evaluation metrics. Finally, we show that agent-generated solutions can be distilled into a reusable program that ranks in the top 10% of baseline methods on unseen data. Our framework offers a scalable way to evaluate AI agents in specialized scientific domains by leveraging and stress-testing existing papers and metrics.
ER -