PreprintarXiv licencearXiv
On attention heads and bilinear forms
Abstract
We study the symmetric and antisymmetric parts of bilinear forms in the attention heads of trained large language models. We introduce an orthogonally invariant profile map from real bilinear forms to a three-dimensional simplex and observe that profiles of trained bilinear forms accumulate near profiles of rank-one bilinear forms. We prove that the symmetric part of a bilinear form in an attention head is the sum of a hyperbolic form and a zero form for a Zariski-dense subset of query-key matrices.
Related
- Exact Fusion of Softmax Attention Heads Can Require Exponentially Many Atoms
- Kernel representations of fractional gradient bilinear forms. Part I. Constant coefficients
- Kernel representations of fractional gradient bilinear forms. Part II. Variable coefficients
- Non-degenerate bilinear forms and left-symmetric structures
- Stein Kernels and Normal Approximation for Log-Concave Bilinear Forms
BibTeX
@misc{odesky2026attention,
title = {{On attention heads and bilinear forms}},
author = {Andrew O'Desky},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2609.22990},
eprint = {2609.22990},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2609.22990},
}
RIS
TY - UNPB
TI - On attention heads and bilinear forms
AU - Andrew O'Desky
PY - 2026
DA - 2026/09/19
PB - arXiv
DO - 10.48550/arxiv.2609.22990
UR - https://arxiv.org/abs/2609.22990
AB - We study the symmetric and antisymmetric parts of bilinear forms in the attention heads of trained large language models. We introduce an orthogonally invariant profile map from real bilinear forms to a three-dimensional simplex and observe that profiles of trained bilinear forms accumulate near profiles of rank-one bilinear forms. We prove that the symmetric part of a bilinear form in an attention head is the sum of a hyperbolic form and a zero form for a Zariski-dense subset of query-key matrices.
ER -