The Birkhoff Geometry of Manifold-Constrained Hyper-Connections: Two Channels, Vertex Viscosity, and Sinkhorn as a Retraction
Abstract
Hyper-connections widen the residual stream of a Transformer to n parallel streams. Their manifold-constrained version (mHC) mixes the streams at each layer with a doubly stochastic matrix, which it computes by Sinkhorn normalization of exponentiated logits. We give a geometric theory of this design on the Birkhoff polytope. First, a doubly stochastic mixer splits the stream into a mean channel, on which mHC is exactly a residual network, and a difference channel, which each layer contracts by its second singular value σ2 ≤ 1 − n minij Hij. Thus the extra width is a fading memory with a horizon of 1/(1−σ2) layers, and among nonnegative mixers only the permutations do not collapse. Second, the Sinkhorn-logit map is a global chart, and its logit gradient is exactly the Fisher-Rao gradient. Thus logit gradient flow follows a squared Fisher-Rao metric, and the straight-through update is exactly entropic mirror descent. Third, under logit gradient flow the logarithm of each entry moves at a rate of at most 4n3|∇ f|∞ ε, where ε is the distance to the nearest permutation. Thus gradient flow approaches and leaves the vertices only at rate 1/t, but mirror descent moves at an exponential rate. Fourth, the local convergence factor of Sinkhorn is σ22, so a fixed iteration budget limits the horizon. Experiments confirm the predicted rates.
Related
- On LCK Geometry of Gauduchon Connections
- Symmetries and Higher-Form Connections in Derived Differential Geometry
- Geometry-Constrained Kolmogorov-Arnold Networks: Learning Edge Geometry via Banach Duality
- Exact Asymptotic Efficiency under zCDP: Diameter-Constrained Information Geometry
- Geometry-Aware Multi-Objective Optimization: Norm-Constrained Oracles with Entropic Weight Updates
BibTeX
@misc{li2026birkhoff,
title = {{The Birkhoff Geometry of Manifold-Constrained Hyper-Connections: Two Channels, Vertex Viscosity, and Sinkhorn as a Retraction}},
author = {Xiaoyu Li and Zhizhou Sha and Chiwun Yang},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.06653},
eprint = {2610.06653},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.06653},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - The Birkhoff Geometry of Manifold-Constrained Hyper-Connections: Two Channels, Vertex Viscosity, and Sinkhorn as a Retraction
AU - Xiaoyu Li
AU - Zhizhou Sha
AU - Chiwun Yang
PY - 2026
DA - 2026/10/05
PB - arXiv
DO - 10.48550/arxiv.2610.06653
UR - https://arxiv.org/abs/2610.06653
AB - Hyper-connections widen the residual stream of a Transformer to n parallel streams. Their manifold-constrained version (mHC) mixes the streams at each layer with a doubly stochastic matrix, which it computes by Sinkhorn normalization of exponentiated logits. We give a geometric theory of this design on the Birkhoff polytope. First, a doubly stochastic mixer splits the stream into a mean channel, on which mHC is exactly a residual network, and a difference channel, which each layer contracts by its second singular value σ_2 ≤ 1 − n min_ij H_ij. Thus the extra width is a fading memory with a horizon of 1/(1−σ_2) layers, and among nonnegative mixers only the permutations do not collapse. Second, the Sinkhorn-logit map is a global chart, and its logit gradient is exactly the Fisher-Rao gradient. Thus logit gradient flow follows a squared Fisher-Rao metric, and the straight-through update is exactly entropic mirror descent. Third, under logit gradient flow the logarithm of each entry moves at a rate of at most 4n^3|∇ f|_∞ ε, where ε is the distance to the nearest permutation. Thus gradient flow approaches and leaves the vertices only at rate 1/t, but mirror descent moves at an exponential rate. Fourth, the local convergence factor of Sinkhorn is σ_2^2, so a fixed iteration budget limits the horizon. Experiments confirm the predicted rates.
ER -