Beyond Shadow Weights: Quantization-Aware Training as Quantized-Endpoint Descent
Abstract
Quantization-aware training (QAT) updates a full-precision shadow weight x but deploys the quantized endpoint Q(x). Existing explanations for QAT largely view its success through the lens of shadow weights: QAT can move x toward flatter basins, gain robustness from quantization-induced oscillations, or balance the shadow loss f(x) against the quantization error |x−Q(x)|2. These perspectives do not directly explain the empirical observation that the deployed endpoint loss f(Q(x)) improves while the shadow loss f(x) does not, and can even increase substantially. In this paper, we offer a different explanation by treating QAT as finite-grid endpoint dynamics. Motivated by the approximate normality of rescaled pretrained weights, we propose an idealized model for the residual phase, which records where each shadow weight sits inside its quantization cell as a fraction of the cell width. This model leads to a crossing law that determines which coordinates cross quantization boundaries after a shadow update. Inspired by the idealized model and signal-imbalance phenomenon in QAT, we further propose QAR (Quantization with Amplified Routing), an algorithmic framework that directly operates on the quantization code. In contrast to QAT, QAR is both theoretically grounded and memory-efficient: it admits feasible-gradient bounds for a family of power amplifiers up to unavoidable finite-grid floors without retaining a full-precision shadow weight copy. Experiments on post-training of large language models provide evidence consistent with the endpoint view and show that QAR can be comparable to or better than QAT with smaller memory cost.
Related
- Quantized Low-Rank Quantum State Tomography: Hyperbolic Quantization and Riemannian Least-Squares Recovery
- Stability and Generalization of Straight-Through Estimators for Training Two-Layer Quantized Neural Networks
- Q-MINO: A Minimal-Norm Method for Quantization-Aware Training
- QUASAR: Lowering the Loss Floor of Quantization-Aware Training with Loss-Aware Reconstruction
- Asymptotic analysis and uniqueness of blowup solutions of non-quantized singular mean field equations
BibTeX
@misc{xu2026beyond,
title = {{Beyond Shadow Weights: Quantization-Aware Training as Quantized-Endpoint Descent}},
author = {Sheng-An Xu and Hanyang Li and Jianhao Ma and Ying Cui},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2609.21834},
eprint = {2609.21834},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2609.21834},
}
RIS
TY - UNPB
TI - Beyond Shadow Weights: Quantization-Aware Training as Quantized-Endpoint Descent
AU - Sheng-An Xu
AU - Hanyang Li
AU - Jianhao Ma
AU - Ying Cui
PY - 2026
DA - 2026/09/18
PB - arXiv
DO - 10.48550/arxiv.2609.21834
UR - https://arxiv.org/abs/2609.21834
AB - Quantization-aware training (QAT) updates a full-precision shadow weight x but deploys the quantized endpoint Q(x). Existing explanations for QAT largely view its success through the lens of shadow weights: QAT can move x toward flatter basins, gain robustness from quantization-induced oscillations, or balance the shadow loss f(x) against the quantization error |x−Q(x)|_2. These perspectives do not directly explain the empirical observation that the deployed endpoint loss f(Q(x)) improves while the shadow loss f(x) does not, and can even increase substantially. In this paper, we offer a different explanation by treating QAT as finite-grid endpoint dynamics. Motivated by the approximate normality of rescaled pretrained weights, we propose an idealized model for the residual phase, which records where each shadow weight sits inside its quantization cell as a fraction of the cell width. This model leads to a crossing law that determines which coordinates cross quantization boundaries after a shadow update. Inspired by the idealized model and signal-imbalance phenomenon in QAT, we further propose QAR (Quantization with Amplified Routing), an algorithmic framework that directly operates on the quantization code. In contrast to QAT, QAR is both theoretically grounded and memory-efficient: it admits feasible-gradient bounds for a family of power amplifiers up to unavoidable finite-grid floors without retaining a full-precision shadow weight copy. Experiments on post-training of large language models provide evidence consistent with the endpoint view and show that QAR can be comparable to or better than QAT with smaller memory cost.
ER -