Physical Muon: Orthogonalization as an Equilibrium Computation
Abstract
Physical neural networks and analog in-memory computing could reduce the energy cost of neural network training. Realizing this potential, however, requires optimizers that combine effective learning with physical implementability. SGD fits local analog updates but struggles on transformers, while Adam family is unstable against analog bias. Muon offers strong training performance, but its Newton--Schulz orthogonalization relies on dense matrix-matrix products. To address this obstacle, we introduce Physical Muon, which computes the orthogonalization as the equilibrium of a continuous-time flow. Random probes approximate the flow using matrix-vector products, reciprocal reads, and local rank-1 writes. To test whether this replacement preserves training performance, we evaluate it on a 10.95M-parameter transformer. The dense flow's mean validation cross-entropy is 0.0085 above Newton--Schulz across nine seeds per method; the probe implementation is 0.0188 above the control across two seeds. Circuit simulations further reproduce the flow dynamics and yield comparable training behavior.
Related
- Training-Loss Guarantees for Muon with Finite-Step Newton--Schulz Orthogonalization
- Muon Learns Facts Better: Understanding the Role of Spectral Orthogonalization
- Polar Switch: Fast Muon Orthogonalization with Polynomial Circuits
- Variable Bound Tightening for Nash Equilibrium Computation in Multiplayer Imperfect-Information Games
- Balance in Inequality: Testing the Physical Equilibrium, Precession, and Structural Stability of Magic Gems
BibTeX
@misc{hao2026physical,
title = {{Physical Muon: Orthogonalization as an Equilibrium Computation}},
author = {Yuren Hao},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2609.37525},
eprint = {2609.37525},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2609.37525},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Physical Muon: Orthogonalization as an Equilibrium Computation
AU - Yuren Hao
PY - 2026
DA - 2026/09/29
PB - arXiv
DO - 10.48550/arxiv.2609.37525
UR - https://arxiv.org/abs/2609.37525
AB - Physical neural networks and analog in-memory computing could reduce the energy cost of neural network training. Realizing this potential, however, requires optimizers that combine effective learning with physical implementability. SGD fits local analog updates but struggles on transformers, while Adam family is unstable against analog bias. Muon offers strong training performance, but its Newton--Schulz orthogonalization relies on dense matrix-matrix products. To address this obstacle, we introduce Physical Muon, which computes the orthogonalization as the equilibrium of a continuous-time flow. Random probes approximate the flow using matrix-vector products, reciprocal reads, and local rank-1 writes. To test whether this replacement preserves training performance, we evaluate it on a 10.95M-parameter transformer. The dense flow's mean validation cross-entropy is 0.0085 above Newton--Schulz across nine seeds per method; the probe implementation is 0.0188 above the control across two seeds. Circuit simulations further reproduce the flow dynamics and yield comparable training behavior.
ER -