{
  "abstract": "Quantum computing remains in the Noisy Intermediate-Scale Quantum (NISQ) era, with performance constrained by noise. Addressing this limitation requires hardware-facing capabilities beyond gate sequences: mid-circuit measurement and classical feedback for quantum error correction (QEC), precise timing for dynamical decoupling (DD), and pulse-level waveform access for calibration. OpenQASM 3 exposes these capabilities through a hardware-level programming interface. Despite rapid progress in large language models (LLMs) for code generation, datasets targeting these advanced features remain lacking. We introduce QASM-Eval, the first comprehensive dataset to train and evaluate LLMs on OpenQASM 3, targeting completion of hardware-facing constructs within a supplied program context. QASM-Eval comprises an expert-curated test set of 1,200 tasks and a training set of over 4,000 tasks, covering classical logic, timing scheduling, pulse control, and complex tasks inspired by quantum-control workflows. An extended verifier automatically checks syntax, quantum measurement distributions, and program timelines. Our evaluation identifies substantial headroom for OpenQASM 3 code generation and significant gains from targeted fine-tuning. Fine-tuned Llama-3-8B outperforms zero-shot GPT-5.6-Terra, while fine-tuned Llama-3-70B achieves 61.17% overall pass@1 and approaches few-shot-augmented GPT-5.6-Terra. QASM-Eval provides a benchmark and training resource for specification-guided code generation over the hardware-facing constructs of OpenQASM 3, supporting the development of LLM assistants for quantum programming. Data and code: https://github.com/fuzhenxiao/QASM-Eval",
  "abstract_shown": true,
  "affiliations": [],
  "article_type": null,
  "arxiv": "2605.30358",
  "author_total": 3,
  "authors": [
    {
      "name": "Zhenxiao Fu"
    },
    {
      "name": "Lei Jiang"
    },
    {
      "name": "Fan Chen"
    }
  ],
  "bibtex": "https://research.draconitic.com/paper/4dc220jz4c6tw.bib",
  "cited_by": 0,
  "doi": "10.48550/arxiv.2605.30358",
  "fulltext_url": null,
  "id": "4dc220jz4c6tw",
  "issns": [],
  "landing_url": "https://arxiv.org/abs/2605.30358",
  "language": null,
  "licence": "CC-BY-4.0",
  "licence_label": "CC BY 4.0",
  "licence_source": "article",
  "licence_url": "http://creativecommons.org/licenses/by/4.0/",
  "metadata_licence": "CC0-1.0",
  "native_subjects": [
    [
      "arxiv",
      "cs.LG"
    ],
    [
      "arxiv",
      "quant-ph"
    ],
    [
      "openalex",
      "1702|Quantum Computing Algorithms and Architecture"
    ]
  ],
  "openalex": "W7163048516",
  "pdf_url": "https://arxiv.org/pdf/2605.30358",
  "pmcid": null,
  "pmid": null,
  "published": "2026-09-25",
  "publisher": null,
  "ris": "https://research.draconitic.com/paper/4dc220jz4c6tw.ris",
  "sources": [
    {
      "first_seen": "2026-10-30T12:24:52Z",
      "native_id": "2605.30358",
      "source": "arxiv",
      "url": "https://arxiv.org/abs/2605.30358"
    }
  ],
  "subjects": [
    {
      "field": "physics",
      "origin": "arxiv:quant-ph",
      "subfield": "quantum"
    }
  ],
  "title": "QASM-Eval: A Dataset to Train and Evaluate LLMs on OpenQASM-3 Beyond Quantum Circuits",
  "title_text": "QASM-Eval: A Dataset to Train and Evaluate LLMs on OpenQASM-3 Beyond Quantum Circuits",
  "type": "preprint",
  "updated_at": "2026-10-30T12:24:52Z",
  "url": "https://research.draconitic.com/paper/4dc220jz4c6tw",
  "venue": "arXiv",
  "versions": []
}