QTestBench: A Test Framework for Evaluating LLM-Generated Quantum Code
Abstract
Large language models (LLMs) have shown strong results in classical code generation and are increasingly used to write the unit tests that safeguard that code. Whether they can do the same for quantum software is less clear, because quantum programs are probabilistic, depend on numerical tolerances, and are judged by domain-specific checkers, so a well-formed test can still disagree with the benchmark's verdict. Existing quantum benchmarks measure whether a model produces a correct solution, but not whether its tests can distinguish correct implementations from faulty ones. We introduce QTestBench, a framework that evaluates LLM-generated quantum test suites against independent benchmark checkers, never treating the tests themselves as ground truth. For each task, QTestBench prompts models to produce syntactic, semantic, and behavioral tests, injects candidate solutions at runtime, and compares each test verdict with the checker's label under three validation cohorts, each answering a distinct diagnostic question. We benchmark five LLMs on eight PennyLane QHack challenges and ten Qiskit HumanEval tasks. Generated suites almost always execute, but discrimination lags behind: the best generator reaches a test quality of 0.71 on QHack and 0.34 on Qiskit HumanEval, so executability alone overstates test quality. Oracle-verified synthetic mutants stabilize bug-detection estimates when natural faulty solutions are scarce, as on three of ten Qiskit tasks that yield none. QTestBench offers a reproducible, cross-framework benchmark for the trustworthiness of LLM-generated quantum tests.
Related
- The stationarity test: a framework for learning quantum many-body systems from their thermal states
- NAKO: A Benchmarking Framework for Evaluating and Diagnosing Neutral-Atom Quantum Compilers
- Evaluating LLM-Generated Preference Distributions
- Analytical framework for computer-generated holography of screen space primitives with point-source integration in the Fourier domain
- Photon triplets for quantum optics generated by phase-matched third-order difference-frequency mixing in a KTiOPO4 bulk crystal pumped at 532 nm
BibTeX
@misc{terki2026qtestbench,
title = {{QTestBench: A Test Framework for Evaluating LLM-Generated Quantum Code}},
author = {Meriem Terki and Nouhaila Innan and Minghao Shao and Muhammad Kashif and Alberto Marchisio and Muhammad Shafique},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.13811},
eprint = {2610.13811},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.13811},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - QTestBench: A Test Framework for Evaluating LLM-Generated Quantum Code
AU - Meriem Terki
AU - Nouhaila Innan
AU - Minghao Shao
AU - Muhammad Kashif
AU - Alberto Marchisio
AU - Muhammad Shafique
PY - 2026
DA - 2026/10/09
PB - arXiv
DO - 10.48550/arxiv.2610.13811
UR - https://arxiv.org/abs/2610.13811
AB - Large language models (LLMs) have shown strong results in classical code generation and are increasingly used to write the unit tests that safeguard that code. Whether they can do the same for quantum software is less clear, because quantum programs are probabilistic, depend on numerical tolerances, and are judged by domain-specific checkers, so a well-formed test can still disagree with the benchmark's verdict. Existing quantum benchmarks measure whether a model produces a correct solution, but not whether its tests can distinguish correct implementations from faulty ones. We introduce QTestBench, a framework that evaluates LLM-generated quantum test suites against independent benchmark checkers, never treating the tests themselves as ground truth. For each task, QTestBench prompts models to produce syntactic, semantic, and behavioral tests, injects candidate solutions at runtime, and compares each test verdict with the checker's label under three validation cohorts, each answering a distinct diagnostic question. We benchmark five LLMs on eight PennyLane QHack challenges and ten Qiskit HumanEval tasks. Generated suites almost always execute, but discrimination lags behind: the best generator reaches a test quality of 0.71 on QHack and 0.34 on Qiskit HumanEval, so executability alone overstates test quality. Oracle-verified synthetic mutants stabilize bug-detection estimates when natural faulty solutions are scarce, as on three of ten Qiskit tasks that yield none. QTestBench offers a reproducible, cross-framework benchmark for the trustworthiness of LLM-generated quantum tests.
ER -