{
  "abstract": "The query and key projections $\\WQ,\\WK$ in attention are almost always trained by Euclidean optimizers with no geometric constraint. We constrain them to the Stiefel manifold and optimize with a Riemannian Adam carrying one scalar second moment per frame---the form of \\citet{becigneul2019}, here extended to the compact, non-Hadamard $\\St(d,r)$ with a tangent projector, step-norm cap, and polar retraction. Four propositions prove steepest descent in the embedded metric, gradient-scale independence, well-conditioning, and exact $\\mathrm{O}(d)$-equivariance. A fifth records that weight decay has \\emph{identically zero} Riemannian gradient on $\\St(d,r)$ ($W{=}WI_r$ lies in the normal space), so decay cannot act on the constrained frames. On a CIFAR-10 patch benchmark at $n{=}10\\mathrm{k}$ this rule gains $\\mathbf{+6.79}$\\,pp over AdamW across 12 paired starts ($t{=}38.33$, $12/12$); earlier fixed-step Riemannian SGD gains $+1.97$\\,pp, of which $+1.69$\\,pp comes from frozen orthonormal initialization alone. The corrected Adam's lead grows with data: $+1.9$\\,pp at $n{=}1\\mathrm{k}$ to $+6.7$\\,pp at $n{=}50\\mathrm{k}$. A 12-seed ablation credits all gain to the scale-free step ($+4.63$\\,pp, $12/12$), nothing to the projector or equivariance; a targeted $\\varepsilon$-sweep causally confirms the mechanism ($-2.6$\\,pp at $\\varepsilon{=}0.1$, $p{<}0.001$). Two five-seed grokking studies confirm the constrained arm does not grok better than the baseline ($p{=}0.019$, A2 wins): the weight-decay exemption has no grokking consequence. A single-seed pilot exploiting this localization achieves the first stable grokking under slingshot conditions---Stiefel + targeted circuit regularization keeps routing-frame isometry error $10^6\\times$ lower than the unconstrained ablation through every collapse.",
  "abstract_shown": true,
  "affiliations": [],
  "article_type": null,
  "arxiv": "2609.19363",
  "author_total": 1,
  "authors": [
    {
      "name": "Rubén Darío Guerrero"
    }
  ],
  "bibtex": "https://research.draconitic.com/paper/8jw6h0t1rssd1.bib",
  "cited_by": 0,
  "doi": "10.48550/arxiv.2609.19363",
  "fulltext_url": null,
  "id": "8jw6h0t1rssd1",
  "issns": [],
  "landing_url": "https://arxiv.org/abs/2609.19363",
  "language": null,
  "licence": "LicenseRef-arXiv-nonexclusive-1.0",
  "licence_label": "arXiv licence",
  "licence_source": "article",
  "licence_url": "http://arxiv.org/licenses/nonexclusive-distrib/1.0/",
  "metadata_licence": "CC0-1.0",
  "native_subjects": [
    [
      "arxiv",
      "cs.LG"
    ],
    [
      "arxiv",
      "cs.NA"
    ],
    [
      "arxiv",
      "math.NA"
    ],
    [
      "openalex",
      "1702|Stochastic Gradient Optimization Techniques"
    ]
  ],
  "openalex": "W7213581174",
  "pdf_url": "https://arxiv.org/pdf/2609.19363",
  "pmcid": null,
  "pmid": null,
  "published": "2026-10-01",
  "publisher": null,
  "ris": "https://research.draconitic.com/paper/8jw6h0t1rssd1.ris",
  "sources": [
    {
      "first_seen": "2026-10-30T12:26:07Z",
      "native_id": "2609.19363",
      "source": "arxiv",
      "url": "https://arxiv.org/abs/2609.19363"
    }
  ],
  "subjects": [
    {
      "field": "mathematics",
      "origin": "arxiv:math.NA",
      "subfield": "numerical-analysis"
    }
  ],
  "title": "Directions That Don't Drift: Stiefel Manifold Routing for Transformer Attention",
  "title_text": "Directions That Don't Drift: Stiefel Manifold Routing for Transformer Attention",
  "type": "preprint",
  "updated_at": "2026-10-30T12:26:07Z",
  "url": "https://research.draconitic.com/paper/8jw6h0t1rssd1",
  "venue": "arXiv",
  "versions": []
}