Variational Quantum Transformer Architecture for Synthetic Language Generation
Abstract
We propose a compact NISQ-compatible quantum transformer architecture for synthetic QNLP sequence modelling. The model preserves the autoregressive next-token interface of a classical transformer, but replaces attention and feed-forward sublayers with variational quantum encoder blocks, connector circuits, decoder blocks and a direct two-qubit measurement readout. Token contexts are angle-encoded into small quantum registers, processed by parallel variational heads and encoder integration circuits and conditioned through decoder ancillae to produce a distribution over a four-token vocabulary. We evaluate several architecture variants on deterministic and lexicographic grammar-generation tasks against a compact classical transformer baseline. The quantum models are trainable end-to-end and learn nontrivial grammar structure, including perfect deterministic generation in individual runs and high lexicographic validity in the strongest variant. The classical baseline remains more accurate and stable and the quantum models are sensitive to initialization. The contribution is therefore not a claim of quantum advantage, but a concrete architecture and evaluation of transformer-inspired QNLP sequence modelling under near-term quantum constraints.
Related
- Variational study of the magnetization plateaus in the spin-1/2 kagome Heisenberg antiferromagnet: An approach from vision transformer neural quantum states
- Dual Branch Transformer with Physical Priors for PSF Acquisition to Enable PSF Guided Image Restoration against Co Phasing Errors in Optical Synthetic Aperture Interferometric Imaging
- Transformer fault diagnosis using an efficient simulation-driven variational quantum classifier with domain-aware feature encoding
- This is SParTA: A modular, efficient and scalable transformer architecture for jet-tagging
- Site-orbit masked twin-branch Transformer for symmetry-constrained crystal generation
BibTeX
@misc{hager2026variational,
title = {{Variational Quantum Transformer Architecture for Synthetic Language Generation}},
author = {Julian Hager and Michael Kölle and Gerhard Stenzel and Tobias Rohe and Jonas Stein and Claudia Linnhoff-Popien},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2609.18565},
eprint = {2609.18565},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2609.18565},
}
RIS
TY - UNPB
TI - Variational Quantum Transformer Architecture for Synthetic Language Generation
AU - Julian Hager
AU - Michael Kölle
AU - Gerhard Stenzel
AU - Tobias Rohe
AU - Jonas Stein
AU - Claudia Linnhoff-Popien
PY - 2026
DA - 2026/09/16
PB - arXiv
DO - 10.48550/arxiv.2609.18565
UR - https://arxiv.org/abs/2609.18565
AB - We propose a compact NISQ-compatible quantum transformer architecture for synthetic QNLP sequence modelling. The model preserves the autoregressive next-token interface of a classical transformer, but replaces attention and feed-forward sublayers with variational quantum encoder blocks, connector circuits, decoder blocks and a direct two-qubit measurement readout. Token contexts are angle-encoded into small quantum registers, processed by parallel variational heads and encoder integration circuits and conditioned through decoder ancillae to produce a distribution over a four-token vocabulary. We evaluate several architecture variants on deterministic and lexicographic grammar-generation tasks against a compact classical transformer baseline. The quantum models are trainable end-to-end and learn nontrivial grammar structure, including perfect deterministic generation in individual runs and high lexicographic validity in the strongest variant. The classical baseline remains more accurate and stable and the quantum models are sensitive to initialization. The contribution is therefore not a claim of quantum advantage, but a concrete architecture and evaluation of transformer-inspired QNLP sequence modelling under near-term quantum constraints.
ER -