CME-SpectrumBench: Can LLMs Analyze Condensed Matter Spectral Data?
Abstract
Large Language Models (LLMs) are presently well assessed by expert-level mathematics and physics benchmarks which focus on theoretical, symbolic reasoning. However, we lack experimental benchmarks that evaluate data analysis in frontier research problems, a capability that future domain-specific foundation models must have. To this end, we present CME-SpectrumBench, an expert-designed benchmark consisting of 190 questions that measures how reliably frontier LLMs analyze research-level spectral scientific data in condensed matter experiments (CME). Using simulated CME spectral data, we prompt models to identify structures from 2D intensity arrays in the face of noise and limited instrumental resolution, a test of dense context reasoning. Models are scored by how accurately their numerical answers approach the ground truth. Under direct query, scores generally pale in comparison to handwritten code due to clear failure modes: models can find simple features but struggle to trace patterns or compute derived quantities, and performance typically degrades with increased data resolution and noise/convolution levels. LLM performance becomes comparable to handwritten code by allowing tool use (access to a Python interpreter) with/without ReAct prompting, in effect mapping questions onto agentic code generation problems. Our findings show that frontier models can serve as research assistants by interfacing with data through code, but still face difficulties processing information directly from large datasets - a key barrier future foundation models must overcome. The code and data are publicly available at https://huggingface.co/datasets/jingene/CMESpectrumBench.
Related
- Quantum geometry phenomena in condensed matter systems
- Ideal interaction dynamics of defects and background fields in condensed matter
- Superinsulation, electric confinement in condensed matter
- Chirality in Condensed Matter: Symmetry, Waves, and Quasiparticles
- Single Dirac Fermions on a Lattice: A Unified View of Chiral and Parity Anomalies in Condensed Matter
BibTeX
@misc{wong2026cmespectrumbench,
title = {{CME-SpectrumBench: Can LLMs Analyze Condensed Matter Spectral Data?}},
author = {Jin Gene Wong and Anjney Midha and Joseph Mathew Tennyson and Wei-Lin Chiang and Ion Stoica and Zhi-Xun Shen},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.24304},
eprint = {2610.24304},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.24304},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - CME-SpectrumBench: Can LLMs Analyze Condensed Matter Spectral Data?
AU - Jin Gene Wong
AU - Anjney Midha
AU - Joseph Mathew Tennyson
AU - Wei-Lin Chiang
AU - Ion Stoica
AU - Zhi-Xun Shen
PY - 2026
DA - 2026/10/16
PB - arXiv
DO - 10.48550/arxiv.2610.24304
UR - https://arxiv.org/abs/2610.24304
AB - Large Language Models (LLMs) are presently well assessed by expert-level mathematics and physics benchmarks which focus on theoretical, symbolic reasoning. However, we lack experimental benchmarks that evaluate data analysis in frontier research problems, a capability that future domain-specific foundation models must have. To this end, we present CME-SpectrumBench, an expert-designed benchmark consisting of 190 questions that measures how reliably frontier LLMs analyze research-level spectral scientific data in condensed matter experiments (CME). Using simulated CME spectral data, we prompt models to identify structures from 2D intensity arrays in the face of noise and limited instrumental resolution, a test of dense context reasoning. Models are scored by how accurately their numerical answers approach the ground truth. Under direct query, scores generally pale in comparison to handwritten code due to clear failure modes: models can find simple features but struggle to trace patterns or compute derived quantities, and performance typically degrades with increased data resolution and noise/convolution levels. LLM performance becomes comparable to handwritten code by allowing tool use (access to a Python interpreter) with/without ReAct prompting, in effect mapping questions onto agentic code generation problems. Our findings show that frontier models can serve as research assistants by interfacing with data through code, but still face difficulties processing information directly from large datasets - a key barrier future foundation models must overcome. The code and data are publicly available at https://huggingface.co/datasets/jingene/CMESpectrumBench.
ER -