Autonomous generalizability auditing of scientific foundation models
- 1Broad Institute of MIT and Harvard
Abstract
Determining where scientific foundation models generalize beyond their development data is essential for reliable use, but aggregate benchmarks obscure performance limits and rigorous audits are difficult to conduct routinely. Here, we introduce AutoSPECTRA, a coding-agent protocol for systematic generalizability auditing. To inform its design, we reviewed 38 studies across five biological domains and evaluated coding agents, which generated hypotheses and empirical analyses but applied inconsistent standards for judging evidence. We then structured AutoSPECTRA around measurable generalization axes, prespecified analyses, controlled comparisons and independent confirmation. Across four biological foundation models, audits linked performance to protein sequence composition and relatedness to pretraining data (ESMFold2), representation of perturbations in training (STATE), effective spatial resolution (CONCH), and residue-level genomic, evolutionary and structural context (AlphaMissense). Agents also evaluated alternative explanations and rejected unsupported hypotheses. AutoSPECTRA provides a systematic approach to discovering and testing the limits of scientific foundation models.
Related
- From Tool State to Scientific Action: Auditing Decision Provenance in Biomedical Research Agents
- Matched-Input Estimates Differ in Sign Across Architectures: Auditing EEG Foundation Models on Motor Imagery
- The Reasoning Paradox in Scientific Sandboxes: Why Extended Reasoning Fails in Autonomous Bioinformatics and How to Fix It.
- Inter-subject correlations and their behavioral associations vary across movies: Implications for generalizability.
- DairySocialTwin: Auditing the video-to-graph evidence chain in cattle social networks
BibTeX
@misc{ektefaie2026autonomous,
title = {{Autonomous generalizability auditing of scientific foundation models}},
author = {Y. Ektefaie and S. Jain and P. Sabeti},
howpublished = {bioRxiv},
year = {2026},
month = nov,
doi = {10.64898/2026.11.03.763979},
url = {https://www.biorxiv.org/content/10.64898/2026.11.03.763979v1},
note = {CC BY},
}
RIS
TY - UNPB
TI - Autonomous generalizability auditing of scientific foundation models
AU - Y. Ektefaie
AU - S. Jain
AU - P. Sabeti
PY - 2026
DA - 2026/11/03
PB - bioRxiv
DO - 10.64898/2026.11.03.763979
UR - https://www.biorxiv.org/content/10.64898/2026.11.03.763979v1
AB - Determining where scientific foundation models generalize beyond their development data is essential for reliable use, but aggregate benchmarks obscure performance limits and rigorous audits are difficult to conduct routinely. Here, we introduce AutoSPECTRA, a coding-agent protocol for systematic generalizability auditing. To inform its design, we reviewed 38 studies across five biological domains and evaluated coding agents, which generated hypotheses and empirical analyses but applied inconsistent standards for judging evidence. We then structured AutoSPECTRA around measurable generalization axes, prespecified analyses, controlled comparisons and independent confirmation. Across four biological foundation models, audits linked performance to protein sequence composition and relatedness to pretraining data (ESMFold2), representation of perturbations in training (STATE), effective spatial resolution (CONCH), and residue-level genomic, evolutionary and structural context (AlphaMissense). Agents also evaluated alternative explanations and rejected unsupported hypotheses. AutoSPECTRA provides a systematic approach to discovering and testing the limits of scientific foundation models.
ER -