Spectral Deflation for Factorization-Free Matrix Filtering in Muon and Semidefinite Programming
Abstract
GPU implementations of the Muon optimizer and of first-order semidefinite programming (SDP) solvers share one computational pattern: a matrix factorization is replaced by a fixed-depth polynomial filter applied after normalization. When a few dominant spectral components carry most of the input's scale, normalization pushes the remaining spectrum toward zero, where the filter is least accurate. We propose spectral deflation: estimate the dominant components, remove them, filter the normalized residual with the unchanged filter, and restore them. Deflation preserves the target matrix function, and we prove that it strictly reduces the finite-step error of the classical Newton--Schulz family. Implemented with batched randomized SVD for Muon and warm-started subspace tracking for ADMM, deflation consistently improves GPT-2 pretraining over the corresponding Muon baselines with both the Newton--Schulz and Polar Express mappings, also in wall-clock time, and lowers the KKT residuals of factorization-free ADMM on large-scale SDPs within a similar projection time.
Related
- A One-Factorization Predictor-Corrector Long-Step Arc-Search Method and a Curvature-Amplified Variant for Semidefinite Programming with a Homogeneous Self-Dual Embedding
- Fast Randomized Entropically Regularized Semidefinite Programming
- Bounds on the Semidefinite Programming Complexity of the Euclidean Ball
- Low-Rank and Lifted Semidefinite Programming for Mixed-Integer Polynomial Power Grid Optimization
- Inexact PDHG for Large-Scale Semidefinite Programming
BibTeX
@misc{sun2026spectral,
title = {{Spectral Deflation for Factorization-Free Matrix Filtering in Muon and Semidefinite Programming}},
author = {Haoran Sun and Shucheng Kang and Heng Yang},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2609.21102},
eprint = {2609.21102},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2609.21102},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Spectral Deflation for Factorization-Free Matrix Filtering in Muon and Semidefinite Programming
AU - Haoran Sun
AU - Shucheng Kang
AU - Heng Yang
PY - 2026
DA - 2026/09/23
PB - arXiv
DO - 10.48550/arxiv.2609.21102
UR - https://arxiv.org/abs/2609.21102
AB - GPU implementations of the Muon optimizer and of first-order semidefinite programming (SDP) solvers share one computational pattern: a matrix factorization is replaced by a fixed-depth polynomial filter applied after normalization. When a few dominant spectral components carry most of the input's scale, normalization pushes the remaining spectrum toward zero, where the filter is least accurate. We propose spectral deflation: estimate the dominant components, remove them, filter the normalized residual with the unchanged filter, and restore them. Deflation preserves the target matrix function, and we prove that it strictly reduces the finite-step error of the classical Newton--Schulz family. Implemented with batched randomized SVD for Muon and warm-started subspace tracking for ADMM, deflation consistently improves GPT-2 pretraining over the corresponding Muon baselines with both the Newton--Schulz and Polar Express mappings, also in wall-clock time, and lowers the KKT residuals of factorization-free ADMM on large-scale SDPs within a similar projection time.
ER -