PreprintarXiv licencearXiv
Geometric knowledge spaces, Attention pairing and Architectures
Abstract
In this paper, we propose a geometric approach to deriving attention mechanisms and exploring potential new transformer architectures. We begin by formalizing the notions of knowledge worlds and attention pairings, then demonstrate how this framework naturally leads to the architectures of standard attention, causal attention, linear attention, tensor attention, and recurrent neural networks. Finally, we introduce several new geometric spaces that may inspire novel neural network architectures.
Related
- Geometric structures, orbit closures and invariant measures in moduli spaces
- Descriptor representations of geometric port-Hamiltonian systems on Hilbert spaces
- A geometric characterization of the Maz'ya-Shaposhnikova limit on metric measure spaces
- Iterated toric descent obstruction and homogeneous spaces over geometric global fields
- Intrinsic Non-Reductive Geometric Invariant Theory and Moduli Spaces of Principal Bundles with Non-Reductive Structure Groups
BibTeX
@misc{xu2026geometric,
title = {{Geometric knowledge spaces, Attention pairing and Architectures}},
author = {Shang Xu},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.29526},
eprint = {2610.29526},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.29526},
}
RIS
TY - UNPB
TI - Geometric knowledge spaces, Attention pairing and Architectures
AU - Shang Xu
PY - 2026
DA - 2026/10/21
PB - arXiv
DO - 10.48550/arxiv.2610.29526
UR - https://arxiv.org/abs/2610.29526
AB - In this paper, we propose a geometric approach to deriving attention mechanisms and exploring potential new transformer architectures. We begin by formalizing the notions of knowledge worlds and attention pairings, then demonstrate how this framework naturally leads to the architectures of standard attention, causal attention, linear attention, tensor attention, and recurrent neural networks. Finally, we introduce several new geometric spaces that may inspire novel neural network architectures.
ER -