A generative pre-trained transformer with Kerr-soliton attention
Abstract
Generative pre-trained transformers (GPTs) have enabled capability-rich language models, but their operation incurs substantial costs in computation, memory, and data movement. A major source of this cost is attention, a nonlinear transformation that computes context-dependent weights from token correlations. Here, we introduce Kerr-soliton attention, harnessing driven-dissipative nonlinear dynamics in a resonator to realize, execute, and validate deep-learning attention in physical hardware. We train a GPT language model using an analytic representation of Kerr-soliton attention and explore generative inference by streaming model-produced inputs through a time-multiplexed soliton ensemble, whose nonlinear evolution transforms the scores into attention weights. We observe high-fidelity agreement between the experimentally produced weights and those predicted by the model. By mapping memory and nonlinear computation to physical dynamics, Kerr-soliton attention points toward hybrid digital-physical systems that reduce data movement in deep learning.
Related
- Inverse reconstruction of dissipative Kerr soliton interactions
- Effect of high odrer dispersion on soliton interaction in coupled Kerr cavities
- Photon-conserving Raman soliton attractors in focusing and defocusing Kerr media
- An all-transistor physical-learning transformer can be trained to perform in-context learning
- Multiscale POD of Transformer Attention Fields: Scale-Selective Analysis via Morlet Scalogram
BibTeX
@misc{williams2026generative,
title = {{A generative pre-trained transformer with Kerr-soliton attention}},
author = {Lindell M. Williams and Yan Jin and Scott B. Papp},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2605.24124},
eprint = {2605.24124},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2605.24124},
}
RIS
TY - UNPB
TI - A generative pre-trained transformer with Kerr-soliton attention
AU - Lindell M. Williams
AU - Yan Jin
AU - Scott B. Papp
PY - 2026
DA - 2026/10/13
PB - arXiv
DO - 10.48550/arxiv.2605.24124
UR - https://arxiv.org/abs/2605.24124
AB - Generative pre-trained transformers (GPTs) have enabled capability-rich language models, but their operation incurs substantial costs in computation, memory, and data movement. A major source of this cost is attention, a nonlinear transformation that computes context-dependent weights from token correlations. Here, we introduce Kerr-soliton attention, harnessing driven-dissipative nonlinear dynamics in a resonator to realize, execute, and validate deep-learning attention in physical hardware. We train a GPT language model using an analytic representation of Kerr-soliton attention and explore generative inference by streaming model-produced inputs through a time-multiplexed soliton ensemble, whose nonlinear evolution transforms the scores into attention weights. We observe high-fidelity agreement between the experimentally produced weights and those predicted by the model. By mapping memory and nonlinear computation to physical dynamics, Kerr-soliton attention points toward hybrid digital-physical systems that reduce data movement in deep learning.
ER -