DiscoverPhysics: Benchmarking LLMs for Out-of-the-Box Scientific Thinking
Abstract
Frontier LLMs now perform strongly across a wide range of physics evaluations, but it is hard to disentangle genuine reasoning from recall of established science. We introduce DiscoverPhysics, an interactive benchmark that asks a LLM agent to discover the laws of motion of a simulated world whose physics deliberately deviates from our own. We construct 22 worlds governed by, among others, screened and fractional-power gravity, multi-species couplings, hidden dark-matter-like particles, non-coordinate-free physics, and time-varying interactions. Each world is generated on demand by an N-body simulator, for which the agent proposes several rounds of experiments, observes raw trajectory data, and ultimately submits both a natural-language explanation of the world's physics and a Python implementation of the inferred law. Because solving a world requires the agent to design informative experiments and revise its hypotheses, the benchmark probes long-horizon reasoning over an experimental history. We evaluate submissions along two complementary axes: trajectory MSE on held-out particles and an LLM-judged explanation score following an expert-written rubric assessing conceptual understanding of each world. Across eleven frontier models, we find that the strongest agents pass only half of the worlds and consistently fail on those where latent structure must be uncovered. Open-source models lag substantially behind commercial models, both in their ability to design informative experiments and in extracting conclusions from the data. We further find that good predictive accuracy does not guarantee high explanation quality and that conceptual understanding depends on hypothesis refinement through well-chosen experiments.
Related
- Scientific capabilities and deployment sustainability of small-scale LLMs in biological wastewater treatment
- IrekoGPT: Turning Structured Pruning into Post-Hoc Slimmable LLMs
- How Do Uncertainty Estimators Align with Hallucination Occurrence in LLMs?
- CodeTD: Topology of Attention Detects Hallucinations in Code LLMs
- Interactive Visualization for Geometry Learning: The Impact of Augmented Reality on Students' Critical Thinking Skills
BibTeX
@misc{wiemann2026discoverphysics,
title = {{DiscoverPhysics: Benchmarking LLMs for Out-of-the-Box Scientific Thinking}},
author = {Matt L. Wiemann and Lindsay M. Smith and Peter Melchior and Siddharth Mishra-Sharma and Andrew Gordon Wilson and Pavel Izmailov and Carolina Cuesta-Lázaro},
howpublished = {arXiv},
year = {2026},
month = aug,
doi = {10.48550/arxiv.2605.26087},
eprint = {2605.26087},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2605.26087},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - DiscoverPhysics: Benchmarking LLMs for Out-of-the-Box Scientific Thinking
AU - Matt L. Wiemann
AU - Lindsay M. Smith
AU - Peter Melchior
AU - Siddharth Mishra-Sharma
AU - Andrew Gordon Wilson
AU - Pavel Izmailov
AU - Carolina Cuesta-Lázaro
PY - 2026
DA - 2026/08/31
PB - arXiv
DO - 10.48550/arxiv.2605.26087
UR - https://arxiv.org/abs/2605.26087
AB - Frontier LLMs now perform strongly across a wide range of physics evaluations, but it is hard to disentangle genuine reasoning from recall of established science. We introduce DiscoverPhysics, an interactive benchmark that asks a LLM agent to discover the laws of motion of a simulated world whose physics deliberately deviates from our own. We construct 22 worlds governed by, among others, screened and fractional-power gravity, multi-species couplings, hidden dark-matter-like particles, non-coordinate-free physics, and time-varying interactions. Each world is generated on demand by an N-body simulator, for which the agent proposes several rounds of experiments, observes raw trajectory data, and ultimately submits both a natural-language explanation of the world's physics and a Python implementation of the inferred law. Because solving a world requires the agent to design informative experiments and revise its hypotheses, the benchmark probes long-horizon reasoning over an experimental history. We evaluate submissions along two complementary axes: trajectory MSE on held-out particles and an LLM-judged explanation score following an expert-written rubric assessing conceptual understanding of each world. Across eleven frontier models, we find that the strongest agents pass only half of the worlds and consistently fail on those where latent structure must be uncovered. Open-source models lag substantially behind commercial models, both in their ability to design informative experiments and in extracting conclusions from the data. We further find that good predictive accuracy does not guarantee high explanation quality and that conceptual understanding depends on hypothesis refinement through well-chosen experiments.
ER -