NEPv Approach for Optimization on Stiefel Manifold with the (2,1)-norm Regularization
Abstract
Row-sparse projection provides a useful tool in machine learning (ML) when it comes to, for example, feature selection, aiming to choose most relevant features for various ML objectives. One way to seek a high quality row-sparse projection is to combine an ML objective, such as the ones for PCA, LDA, and OCCA, with the matrix (2,1)-norm regularization which is nonsmooth. Such combinations result in challenging optimization problems on the Stiefel manifold that need to be solved efficiently. In this paper, a unifying NEPv framework is established to efficiently deal with optimization on the Stiefel manifold with the (2,1)-norm regularization. The effect of the (2,1)-norm regularization is also investigated. The wide applicability of the framework is demonstrated through the combinations of common learning objectives in today's data science applications with the (2,1)-norm regularization. Numerical experiments are presented to illustrate the use of the NEPv approach and to gain insights as to what a proper regularizing parameter should have in real-world applications.
Related
- PLADOS: A Parameter-free Landing Algorithm for Decentralized Optimization on the Stiefel Manifold
- Infeasible Momentum Orthogonalized Methods for Optimization on the Stiefel Manifold
- Retraction-Free Nonsmooth Optimization on the Stiefel Manifold: Subgradient and Proximal Landing Methods
- A Safeguarded Transport-Extrapolated Proximal Gradient Method on the Stiefel Manifold
- Escaping the Capacity Ceiling: Routing on the Stiefel Manifold for Bilinear SPD Layers
BibTeX
@misc{li2026nepv,
title = {{NEPv Approach for Optimization on Stiefel Manifold with the $(2,1)$-norm Regularization}},
author = {Ren-Cang Li and Li Wang and Lei-Hong Zhang and Zhaojun Bai},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2609.26675},
eprint = {2609.26675},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2609.26675},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - NEPv Approach for Optimization on Stiefel Manifold with the (2,1)-norm Regularization
AU - Ren-Cang Li
AU - Li Wang
AU - Lei-Hong Zhang
AU - Zhaojun Bai
PY - 2026
DA - 2026/09/22
PB - arXiv
DO - 10.48550/arxiv.2609.26675
UR - https://arxiv.org/abs/2609.26675
AB - Row-sparse projection provides a useful tool in machine learning (ML) when it comes to, for example, feature selection, aiming to choose most relevant features for various ML objectives. One way to seek a high quality row-sparse projection is to combine an ML objective, such as the ones for PCA, LDA, and OCCA, with the matrix (2,1)-norm regularization which is nonsmooth. Such combinations result in challenging optimization problems on the Stiefel manifold that need to be solved efficiently. In this paper, a unifying NEPv framework is established to efficiently deal with optimization on the Stiefel manifold with the (2,1)-norm regularization. The effect of the (2,1)-norm regularization is also investigated. The wide applicability of the framework is demonstrated through the combinations of common learning objectives in today's data science applications with the (2,1)-norm regularization. Numerical experiments are presented to illustrate the use of the NEPv approach and to gain insights as to what a proper regularizing parameter should have in real-world applications.
ER -