PreprintarXiv licencearXiv
Pion: A Spectrum-Preserving Optimizer via Orthogonal Equivalence Transformation
Abstract
We introduce Pion, a spectrum-preserving optimizer for large language model (LLM) training based on orthogonal equivalence transformation. Unlike additive optimizers such as Adam and Muon, Pion updates each weight matrix through left and right orthogonal transformations, preserving its singular values throughout training. This yields an optimization mechanism that modulates the geometry of weight matrices while keeping their spectral norm fixed. We derive the Pion update rule, systematically examine its design choices, and analyze its convergence behavior along with several key properties. Empirical results show that Pion offers a stable and competitive alternative to standard optimizers for both LLM pretraining and finetuning.
Related
- Correlated Gaussian Moments Determine Polynomials up to Orthogonal Equivalence
- Local unitary equivalence of orthogonal arrays and related linear codes
- An Exploration of Lacunary Orthogonal Polynomials Associated with the Geronimus Transformation
- A unified approach via Geronimus transformation to various types of orthogonal polynomials
- Embedding Linear Codes into Optimal Self-Orthogonal Codes Based on Constraint-Preserving Genetic Algorithms
BibTeX
@misc{shi2026pion,
title = {{Pion: A Spectrum-Preserving Optimizer via Orthogonal Equivalence Transformation}},
author = {Kexuan Shi and Hanxuan Li and Zeju Qiu and Yandong Wen and Simon Buchholz and Weiyang Liu},
howpublished = {arXiv},
year = {2026},
month = nov,
doi = {10.48550/arxiv.2605.12492},
eprint = {2605.12492},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2605.12492},
}
RIS
TY - UNPB
TI - Pion: A Spectrum-Preserving Optimizer via Orthogonal Equivalence Transformation
AU - Kexuan Shi
AU - Hanxuan Li
AU - Zeju Qiu
AU - Yandong Wen
AU - Simon Buchholz
AU - Weiyang Liu
PY - 2026
DA - 2026/11/08
PB - arXiv
DO - 10.48550/arxiv.2605.12492
UR - https://arxiv.org/abs/2605.12492
AB - We introduce Pion, a spectrum-preserving optimizer for large language model (LLM) training based on orthogonal equivalence transformation. Unlike additive optimizers such as Adam and Muon, Pion updates each weight matrix through left and right orthogonal transformations, preserving its singular values throughout training. This yields an optimization mechanism that modulates the geometry of weight matrices while keeping their spectral norm fixed. We derive the Pion update rule, systematically examine its design choices, and analyze its convergence behavior along with several key properties. Empirical results show that Pion offers a stable and competitive alternative to standard optimizers for both LLM pretraining and finetuning.
ER -