Science sandboxes measure the scientific capability of AI agents
Abstract
Scientific progress depends not only on finding solutions, but on learning the rules that explain why they work and using that understanding to design better experiments. We introduce science sandboxes, a framework for studying this capability in AI agents through repeated cycles of experimentation, feedback, and hypothesis revision. Science sandboxes invite an agent to query the natural world in different ways, ranging from "wet" physical experiments, to "damp" predictive models trained on empirical data, to "dry" invented rules. By establishing a common experimental loop and a protocol for evaluating agents within it, science sandboxes allow assessment of both quantitative performance on specific metrics and qualitative scientific reasoning, across a spectrum of empirical verifiability. Here, we instantiate this framework in two biological settings, models of regulatory genomics and protein fitness prediction, and examine the capabilities of frontier agents. Across these settings, we could see when agents successfully optimized a quantitative metric without understanding the rules underlying the system. In particular, their scientific reasoning deteriorated when they encountered systems whose rules fell outside familiar biological priors. By highlighting such failure modes, science sandboxes make the frontier of scientific capability measurable and provide a controlled setting in which to study and ultimately expand it.
Related
- The Reasoning Paradox in Scientific Sandboxes: Why Extended Reasoning Fails in Autonomous Bioinformatics and How to Fix It.
- Scientific criteria and guidelines for identifying, reporting and interpreting omics data – transcriptomics, proteomics, metabolomics and lipidomics – in meat science research
- How science gets by with a little help from the Beatles: Detecting cultural wordplay in scientific literature using large language models
- From Tool State to Scientific Action: Auditing Decision Provenance in Biomedical Research Agents
- Self-Generated Goals in the Age of Foundation Models: Artificial Autotelic Agents and Their Implications for Cognitive Science
BibTeX
@misc{rao2026science,
title = {{Science sandboxes measure the scientific capability of AI agents}},
author = {Arya S. Rao and Rodrigo I. Castro and Sager J. Gosai and Kenneth B. Hsu and Yasha Ektefaie and Shantanu Singh and Sangeeta N. Bhatia and Steven K. Reilly and Ryan Tewhey and Eric S. Lander and Pardis C. Sabeti},
howpublished = {arXiv},
year = {2026},
month = aug,
doi = {10.48550/arxiv.2608.30165},
eprint = {2608.30165},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2608.30165},
note = {CC BY-NC-ND 4.0},
}
RIS
TY - UNPB
TI - Science sandboxes measure the scientific capability of AI agents
AU - Arya S. Rao
AU - Rodrigo I. Castro
AU - Sager J. Gosai
AU - Kenneth B. Hsu
AU - Yasha Ektefaie
AU - Shantanu Singh
AU - Sangeeta N. Bhatia
AU - Steven K. Reilly
AU - Ryan Tewhey
AU - Eric S. Lander
AU - Pardis C. Sabeti
PY - 2026
DA - 2026/08/31
PB - arXiv
DO - 10.48550/arxiv.2608.30165
UR - https://arxiv.org/abs/2608.30165
AB - Scientific progress depends not only on finding solutions, but on learning the rules that explain why they work and using that understanding to design better experiments. We introduce science sandboxes, a framework for studying this capability in AI agents through repeated cycles of experimentation, feedback, and hypothesis revision. Science sandboxes invite an agent to query the natural world in different ways, ranging from "wet" physical experiments, to "damp" predictive models trained on empirical data, to "dry" invented rules. By establishing a common experimental loop and a protocol for evaluating agents within it, science sandboxes allow assessment of both quantitative performance on specific metrics and qualitative scientific reasoning, across a spectrum of empirical verifiability. Here, we instantiate this framework in two biological settings, models of regulatory genomics and protein fitness prediction, and examine the capabilities of frontier agents. Across these settings, we could see when agents successfully optimized a quantitative metric without understanding the rules underlying the system. In particular, their scientific reasoning deteriorated when they encountered systems whose rules fell outside familiar biological priors. By highlighting such failure modes, science sandboxes make the frontier of scientific capability measurable and provide a controlled setting in which to study and ultimately expand it.
ER -