Infeasible Momentum Orthogonalized Methods for Optimization on the Stiefel Manifold
Abstract
We study deterministic and stochastic optimization over the Stiefel manifold, explicitly allowing iterates to deviate from the manifold rather than enforcing feasibility via retraction at each step. We introduce Landom, a Landing-type method that constructs momentum in the relative-gradient direction, maps this update to its compact polar factor, and combines the resulting tangential step with a step along the negative gradient of the orthogonality penalty to improve feasibility. Under operator-to-nuclear smoothness, Landom achieves an O(T−1/2) bound on the averaged nuclear norm of the relative gradient with exact gradients, and an O(T−1/4) bound in expectation with stochastic momentum, bounded variance, and a fixed batch size. Our analysis also provides explicit pathwise bounds on the orthogonality residual, including its dependence on the step size and initial infeasibility. To address the coupling between objective and feasibility correction in Landom, we further propose Landom+, a two-timescale variant in which the tangential and normal step sizes are decoupled. By keeping the normal step size fixed, Landom+ achieves tighter control over the orthogonality residual following an initial transient while maintaining the respective convergence rates for stationarity. Numerical experiments across a range of applications illustrate that the proposed algorithms remain competitive with established manifold optimization approaches.
Related
- ProLand: A New Infeasible Manifold Optimization Method for LoRA Fine-Tuning
- Nonsmooth Optimization via Orthogonalized Momentum
- Sharpness-aware Orthogonalized Momentum Algorithms
- Retraction-Free Nonsmooth Optimization on the Stiefel Manifold: Subgradient and Proximal Landing Methods
- Optimal Momentum Methods for Stochastic Multilevel Compositional Optimization
BibTeX
@misc{zeng2026infeasible,
title = {{Infeasible Momentum Orthogonalized Methods for Optimization on the Stiefel Manifold}},
author = {Yuxuan Zeng and Qi Deng and Dongdong Ge and Yinyu Ye},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.41406},
eprint = {2610.41406},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.41406},
}
RIS
TY - UNPB
TI - Infeasible Momentum Orthogonalized Methods for Optimization on the Stiefel Manifold
AU - Yuxuan Zeng
AU - Qi Deng
AU - Dongdong Ge
AU - Yinyu Ye
PY - 2026
DA - 2026/10/30
PB - arXiv
DO - 10.48550/arxiv.2610.41406
UR - https://arxiv.org/abs/2610.41406
AB - We study deterministic and stochastic optimization over the Stiefel manifold, explicitly allowing iterates to deviate from the manifold rather than enforcing feasibility via retraction at each step. We introduce Landom, a Landing-type method that constructs momentum in the relative-gradient direction, maps this update to its compact polar factor, and combines the resulting tangential step with a step along the negative gradient of the orthogonality penalty to improve feasibility. Under operator-to-nuclear smoothness, Landom achieves an O(T^−1/2) bound on the averaged nuclear norm of the relative gradient with exact gradients, and an O(T^−1/4) bound in expectation with stochastic momentum, bounded variance, and a fixed batch size. Our analysis also provides explicit pathwise bounds on the orthogonality residual, including its dependence on the step size and initial infeasibility. To address the coupling between objective and feasibility correction in Landom, we further propose Landom+, a two-timescale variant in which the tangential and normal step sizes are decoupled. By keeping the normal step size fixed, Landom+ achieves tighter control over the orthogonality residual following an initial transient while maintaining the respective convergence rates for stationarity. Numerical experiments across a range of applications illustrate that the proposed algorithms remain competitive with established manifold optimization approaches.
ER -