ManifoldFlow: Spectrum-Learnable Stiefel Optimization via SPD Relaxation
Abstract
Orthogonal and Stiefel constraints are widely used in neural network optimization to control weight geometry, improve conditioning, and stabilize signal propagation. Yet standard Stiefel optimization imposes a hidden restriction: while learning the directions of a linear map, it fixes every singular value to one, preventing the model from learning how strongly different directions should be amplified or suppressed. We introduce ManifoldFlow, a spectrum-learnable extension of Stiefel optimization that separates directional geometry from spectral scaling through W=QS1/2, where Q⊤ Q=I and S≻0. Beyond simply making the spectrum trainable, ManifoldFlow uses the gradient component rejected by the Stiefel constraint as a signal for how the spectrum should change, accumulates this signal across aligned tangent spaces, and jointly updates the orthonormal basis and a bounded SPD spectrum. This construction preserves exact spectral interpretability because W⊤ W=S, while recovering standard Stiefel optimization when S=I. We establish the geometry of the rejected gradient, characterize optimization over the resulting Stiefel--SPD product space, and derive how the learned spectrum affects conditioning, signal propagation, and spectral complexity. Across diverse tasks, ManifoldFlow consistently outperforms matched fixed-spectrum Stiefel baselines, with particularly strong gains in recurrent language modeling. Our code is available at https://github.com/Hik289/manifold_flow.git.
Related
- PLADOS: A Parameter-free Landing Algorithm for Decentralized Optimization on the Stiefel Manifold
- NEPv Approach for Optimization on Stiefel Manifold with the (2,1)-norm Regularization
- Infeasible Momentum Orthogonalized Methods for Optimization on the Stiefel Manifold
- Retraction-Free Nonsmooth Optimization on the Stiefel Manifold: Subgradient and Proximal Landing Methods
- Recursive Gradient Retraction Methods for Stochastic Nonconvex Optimization on Stiefel Manifolds
BibTeX
@misc{yi2026manifoldflow,
title = {{ManifoldFlow: Spectrum-Learnable Stiefel Optimization via SPD Relaxation}},
author = {Haiwen Yi and Xinyuan Song},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2607.04535},
eprint = {2607.04535},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2607.04535},
}
RIS
TY - UNPB
TI - ManifoldFlow: Spectrum-Learnable Stiefel Optimization via SPD Relaxation
AU - Haiwen Yi
AU - Xinyuan Song
PY - 2026
DA - 2026/10/15
PB - arXiv
DO - 10.48550/arxiv.2607.04535
UR - https://arxiv.org/abs/2607.04535
AB - Orthogonal and Stiefel constraints are widely used in neural network optimization to control weight geometry, improve conditioning, and stabilize signal propagation. Yet standard Stiefel optimization imposes a hidden restriction: while learning the directions of a linear map, it fixes every singular value to one, preventing the model from learning how strongly different directions should be amplified or suppressed. We introduce ManifoldFlow, a spectrum-learnable extension of Stiefel optimization that separates directional geometry from spectral scaling through W=QS^1/2, where Q^⊤ Q=I and S≻0. Beyond simply making the spectrum trainable, ManifoldFlow uses the gradient component rejected by the Stiefel constraint as a signal for how the spectrum should change, accumulates this signal across aligned tangent spaces, and jointly updates the orthonormal basis and a bounded SPD spectrum. This construction preserves exact spectral interpretability because W^⊤ W=S, while recovering standard Stiefel optimization when S=I. We establish the geometry of the rejected gradient, characterize optimization over the resulting Stiefel--SPD product space, and derive how the learned spectrum affects conditioning, signal propagation, and spectral complexity. Across diverse tasks, ManifoldFlow consistently outperforms matched fixed-spectrum Stiefel baselines, with particularly strong gains in recurrent language modeling. Our code is available at https://github.com/Hik289/manifold_flow.git.
ER -