Convergence of Rotation-based Matrix Optimizers: A Unified Analysis of SOAP, Conda, and SPlus
Abstract
In this work, we develop a unified theoretical framework for analyzing the convergence of rotation-based matrix optimizers, which apply orthogonal transformations to map the momentum into a rotated space, perform coordinate-wise or normalized updates there, and then rotate the updates back. Our framework encompasses prominent matrix optimizers, including SOAP, Conda, and truncated SPlus, as well as matrix-parameterized Adam as a special case. As our main result, we establish, for the first time, the convergence rate of SOAP with sharp dimensional dependence, as well as the convergence rate of Adam measured by the nuclear norm. Our framework also covers a more general variant of rotation-based optimizers that allows arbitrary orthogonal rotation matrices, allowing these matrices to depend on the current stochastic gradient at each iteration. Technically, our framework leverages a row-column trace-control argument that converts elementwise bounds into bounds on two diagonal control matrices, thereby improving the dimension dependence of the resulting bound.
Related
- Rotation groups, mediangle graphs, and periagroups: A unified point of view on Coxeter groups and graph products of groups
- On the Approximation of the Unitary Operator Group Associated with a Rotation Matrix and Its Applications to Abstract Hyperbolic Equations
- On the Unified Generalized q-Stirling Matrices and Their Matrix Decompositions
- Asynchronous Jacobi and randomized Gauss--Seidel methods in shared and distributed memory: A unified convergence-rate analysis
- A Unified Framework for Wasserstein Convergence of ULMC Methods beyond Log-Concavity: Old and New
BibTeX
@misc{sun2026convergence,
title = {{Convergence of Rotation-based Matrix Optimizers: A Unified Analysis of SOAP, Conda, and SPlus}},
author = {Yiwen Sun and Huan Li and Zhouchen Lin},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2609.24522},
eprint = {2609.24522},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2609.24522},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Convergence of Rotation-based Matrix Optimizers: A Unified Analysis of SOAP, Conda, and SPlus
AU - Yiwen Sun
AU - Huan Li
AU - Zhouchen Lin
PY - 2026
DA - 2026/09/21
PB - arXiv
DO - 10.48550/arxiv.2609.24522
UR - https://arxiv.org/abs/2609.24522
AB - In this work, we develop a unified theoretical framework for analyzing the convergence of rotation-based matrix optimizers, which apply orthogonal transformations to map the momentum into a rotated space, perform coordinate-wise or normalized updates there, and then rotate the updates back. Our framework encompasses prominent matrix optimizers, including SOAP, Conda, and truncated SPlus, as well as matrix-parameterized Adam as a special case. As our main result, we establish, for the first time, the convergence rate of SOAP with sharp dimensional dependence, as well as the convergence rate of Adam measured by the nuclear norm. Our framework also covers a more general variant of rotation-based optimizers that allows arbitrary orthogonal rotation matrices, allowing these matrices to depend on the current stochastic gradient at each iteration. Technically, our framework leverages a row-column trace-control argument that converts elementwise bounds into bounds on two diagonal control matrices, thereby improving the dimension dependence of the resulting bound.
ER -