Riemannian Structure and Optimization for a Class of Low-Parametric Orthogonal Matrices
Abstract
In this paper, we are concerned with matrices formed by block-diagonal factors interleaved with fixed permutations -- a flexible family of structured matrices. This class has recently drawn interest in deep learning architectures for its balanced expressivity-efficiency trade-off, yet efficient computational strategies for working with it remain to be found. We approach this problem through Riemannian geometry and examine under what conditions this class admits a smooth manifold structure. For the practically important case of orthogonal two-factor matrices, we derive the essential Riemannian tools and propose efficient algorithms for their implementation. The algorithms leverage automatic differentiation, support parameter sharing within each factor, and avoid explicit dense matrix construction. We test them within the Riemannian optimization framework on the best matrix approximation problem and for parameter-efficient fine-tuning of large language models. Beyond the two-factor setting, we study the geometric and matrix-theoretic properties of factorizations with a larger number of block-diagonal factors.
Related
- Riemannian Optimization for Hadamard Products of Low-Rank Matrices
- Computational study of Proper Orthogonal Decomposition methods for parametric approximations
- Adaptive Local Proper Orthogonal Decomposition via Graph Partitioning and Provable Error-Bounded Lossy Compression for Parametric Series in Finite-Dimensional Function Spaces
- Fast tensor transforms and ring-valued orthogonal matrices: an application to cryptography
- Characteristic identities for orthogonal quantum matrices: examples
BibTeX
@misc{aliev2026riemannian,
title = {{Riemannian Structure and Optimization for a Class of Low-Parametric Orthogonal Matrices}},
author = {Ali Aliev and Maxim Rakhuba},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2609.27982},
eprint = {2609.27982},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2609.27982},
}
RIS
TY - UNPB
TI - Riemannian Structure and Optimization for a Class of Low-Parametric Orthogonal Matrices
AU - Ali Aliev
AU - Maxim Rakhuba
PY - 2026
DA - 2026/09/23
PB - arXiv
DO - 10.48550/arxiv.2609.27982
UR - https://arxiv.org/abs/2609.27982
AB - In this paper, we are concerned with matrices formed by block-diagonal factors interleaved with fixed permutations -- a flexible family of structured matrices. This class has recently drawn interest in deep learning architectures for its balanced expressivity-efficiency trade-off, yet efficient computational strategies for working with it remain to be found. We approach this problem through Riemannian geometry and examine under what conditions this class admits a smooth manifold structure. For the practically important case of orthogonal two-factor matrices, we derive the essential Riemannian tools and propose efficient algorithms for their implementation. The algorithms leverage automatic differentiation, support parameter sharing within each factor, and avoid explicit dense matrix construction. We test them within the Riemannian optimization framework on the best matrix approximation problem and for parameter-efficient fine-tuning of large language models. Beyond the two-factor setting, we study the geometric and matrix-theoretic properties of factorizations with a larger number of block-diagonal factors.
ER -