QEVOLVE-Bench: A Seed Benchmark for Quantum SDK Evolution and Repair Planning
Abstract
Quantum software is increasingly built on fast-moving Python SDKs such as Qiskit, PennyLane, and Cirq. When these SDKs evolve, user programs can fail because execution helpers are removed, import paths change, simulator abstractions shift, device names are deprecated, or circuit export interfaces are revised. Although some failures surface as simple missing-symbol errors, repairing them is not only a syntactic problem: developers must identify the replacement workflow, preserve the quantum-facing intent, and validate behavior under pinned framework versions. This paper presents QEVOLVE-Bench, a small executable seed benchmark for quantum SDK evolution. The current public release contains 14 controlled migration tasks across Qiskit, PennyLane, and Cirq. Each task is a self-contained Python project with pinned dependencies, failing and passing outputs, pytest acceptance checks, metadata, and benchmark notes. The artifact also includes a manifest, sanity checker, representative smoke-test logs, Zenodo archive, screencast, and a lightweight QEVOLVE-Agent prototype that converts task evidence into structured repair proposals. QEVOLVE-Bench is intentionally not a statistically powered benchmark; instead, it is an extensible seed artifact that lowers the setup cost for studying automated repair, LLM-based software engineering, and maintenance of quantum SDK-based projects.
Related
- OpenGray: a benchmark for evaluating the decisions of radiotherapy planning agents
- QC-Stark: A Multi-Task Benchmark Revealing Capability Dissociations in LLMs Evaluated on Quantum Computing Tasks
- Interaction-driven dynamics in graphene flakes as a benchmark for quantum simulation
- Generalized Svetlichny Inequalities as a Cross-Platform Benchmark for Quantum Processors
- Toward Circuit-Aware Mutation Testing for Quantum Programs: Exact Visibility and Execution Planning
BibTeX
@misc{bhatia2026qevolvebench,
title = {{QEVOLVE-Bench: A Seed Benchmark for Quantum SDK Evolution and Repair Planning}},
author = {Krishna Bhatia and Gautami Sanjay Naik},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2609.29492},
eprint = {2609.29492},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2609.29492},
}
RIS
TY - UNPB
TI - QEVOLVE-Bench: A Seed Benchmark for Quantum SDK Evolution and Repair Planning
AU - Krishna Bhatia
AU - Gautami Sanjay Naik
PY - 2026
DA - 2026/09/01
PB - arXiv
DO - 10.48550/arxiv.2609.29492
UR - https://arxiv.org/abs/2609.29492
AB - Quantum software is increasingly built on fast-moving Python SDKs such as Qiskit, PennyLane, and Cirq. When these SDKs evolve, user programs can fail because execution helpers are removed, import paths change, simulator abstractions shift, device names are deprecated, or circuit export interfaces are revised. Although some failures surface as simple missing-symbol errors, repairing them is not only a syntactic problem: developers must identify the replacement workflow, preserve the quantum-facing intent, and validate behavior under pinned framework versions. This paper presents QEVOLVE-Bench, a small executable seed benchmark for quantum SDK evolution. The current public release contains 14 controlled migration tasks across Qiskit, PennyLane, and Cirq. Each task is a self-contained Python project with pinned dependencies, failing and passing outputs, pytest acceptance checks, metadata, and benchmark notes. The artifact also includes a manifest, sanity checker, representative smoke-test logs, Zenodo archive, screencast, and a lightweight QEVOLVE-Agent prototype that converts task evidence into structured repair proposals. QEVOLVE-Bench is intentionally not a statistically powered benchmark; instead, it is an extensible seed artifact that lowers the setup cost for studying automated repair, LLM-based software engineering, and maintenance of quantum SDK-based projects.
ER -