Unmasking shortcut solutions of coding agents on partial differential equations
Abstract
Coding agents now write simulation programs, but benchmarks usually judge only whether their output matches a reference. In manufactured-solution tasks the data determine the exact solution, so a matching output need not come from a solver. We introduce PDEViBench, a benchmark of 290 tasks in six equation families that traces each passing program to the computation behind its output and reruns unclear programs on changed data. A Codex agent (GPT-5.2) passed 80% of 1,450 runs, yet about one pass in ten returned the exact solution without solving the equation. Counting only solved passes changed which task features made problems seem hard and reversed the ranking of two prompts. Once the prompt forbade the shortcut, every pass was a numerical solve, and on tasks without a closed-form solution passing programs stayed correct on changed data. Scientific coding benchmarks should check how each answer was computed as well as whether it matches.
Related
- AKAPINN: Adaptive Kolmogorov-Arnold Physics-Informed Neural Networks for approximating solutions to quasilinear partial differential equations
- An Adaptive Transferable Neural Network for Partial Differential Equations with Anisotropic or Sharp Solutions
- Entire solutions of Fermat-Type Partial Differential-Difference equations in higher dimensions
- Partial Differential Equations: Overview, Applications, and Numerical Solutions
- Functional Analysis and Partial Differential Equations in Spectral Barron Spaces
BibTeX
@misc{linghu2026unmasking,
title = {{Unmasking shortcut solutions of coding agents on partial differential equations}},
author = {Jiale Linghu and Yangshuai Wang},
howpublished = {arXiv},
year = {2026},
month = nov,
doi = {10.48550/arxiv.2611.12308},
eprint = {2611.12308},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2611.12308},
}
RIS
TY - UNPB
TI - Unmasking shortcut solutions of coding agents on partial differential equations
AU - Jiale Linghu
AU - Yangshuai Wang
PY - 2026
DA - 2026/11/09
PB - arXiv
DO - 10.48550/arxiv.2611.12308
UR - https://arxiv.org/abs/2611.12308
AB - Coding agents now write simulation programs, but benchmarks usually judge only whether their output matches a reference. In manufactured-solution tasks the data determine the exact solution, so a matching output need not come from a solver. We introduce PDEViBench, a benchmark of 290 tasks in six equation families that traces each passing program to the computation behind its output and reruns unclear programs on changed data. A Codex agent (GPT-5.2) passed 80% of 1,450 runs, yet about one pass in ten returned the exact solution without solving the equation. Counting only solved passes changed which task features made problems seem hard and reversed the ranking of two prompts. Once the prompt forbade the shortcut, every pass was a numerical solve, and on tasks without a closed-form solution passing programs stayed correct on changed data. Scientific coding benchmarks should check how each answer was computed as well as whether it matches.
ER -