How Much Does Random Rotation Help GPTQ? Sharp Worst-Case Rate Gaps in the High-Rate Regime
Abstract
GPTQ is Babai's nearest-plane algorithm on the lattice of the Cholesky factor of the layer's input second moment Σ. In the high-rate, entropy-coded regime it needs 1/2log2(AM/GM) of its Cholesky pivots more bits than WaterSIC, which is within $0.255$ bit of the rate-distortion limit for every Σ; random rotations close most of this gap in practice, and Ordentlich and Polyanskiy asked for the worst-case gap between rotated GPTQ and WaterSIC. We show that after a Haar rotation the k-th pivot converges almost surely, uniformly in k≥δn and over spectra ν of bounded condition number, to 1/Sν(−k/n), a classical formula from the theory of Lyapunov exponents, and use it to answer the question. Over spectra with condition number κ the worst rotated gap is 1/2log2κ−log2lnκ+1/2−1/ln2+o(1) bits, attained only by two-point spectra, just 1/2log2lnκ+O(1) bits below the worst case without rotation, so rotation cannot replace waterfilling in general. It nearly can on real layers: rotation cuts the cost of an η-fraction of outlier eigenvalues of size μ from about ημ to η2μ/2 to leading order, but cannot remove spectral spread on a logarithmic scale; for non-Gaussian weights it also adds a weight-side rate cost, which we characterize exactly under a standard high-rate assumption. On $96$ Hessians of Qwen3 and Llama-3.1 the rotated gap has median $0.04$ bit and maximum $0.36$ bit, against $2−−5$ bits in the worst case for the same condition numbers, and the limit matches it to within $0.009$ bit.
Related
- Universality for the global spectrum of random inner-product kernel matrices in the polynomial regime
- Sharp mean-field estimates for diffusive log/Riesz gases in the Hilbert--Schmidt regime
- Maximum degree for random geometric graphs in the connectivity regime and beyond
- Local limit Theorem for Directed polymers in a random environment in the full weak disorder regime
- Wasserstein mixing of a systematic-scan random rotation sampler
BibTeX
@misc{li2026much,
title = {{How Much Does Random Rotation Help GPTQ? Sharp Worst-Case Rate Gaps in the High-Rate Regime}},
author = {Runguo Li},
howpublished = {arXiv},
year = {2026},
month = nov,
doi = {10.48550/arxiv.2611.01669},
eprint = {2611.01669},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2611.01669},
}
RIS
TY - UNPB
TI - How Much Does Random Rotation Help GPTQ? Sharp Worst-Case Rate Gaps in the High-Rate Regime
AU - Runguo Li
PY - 2026
DA - 2026/11/01
PB - arXiv
DO - 10.48550/arxiv.2611.01669
UR - https://arxiv.org/abs/2611.01669
AB - GPTQ is Babai's nearest-plane algorithm on the lattice of the Cholesky factor of the layer's input second moment Σ. In the high-rate, entropy-coded regime it needs 1/2log_2(AM/GM) of its Cholesky pivots more bits than WaterSIC, which is within $0.255$ bit of the rate-distortion limit for every Σ; random rotations close most of this gap in practice, and Ordentlich and Polyanskiy asked for the worst-case gap between rotated GPTQ and WaterSIC. We show that after a Haar rotation the k-th pivot converges almost surely, uniformly in k≥δn and over spectra ν of bounded condition number, to 1/S_ν(−k/n), a classical formula from the theory of Lyapunov exponents, and use it to answer the question. Over spectra with condition number κ the worst rotated gap is 1/2log_2κ−log_2lnκ+1/2−1/ln2+o(1) bits, attained only by two-point spectra, just 1/2log_2lnκ+O(1) bits below the worst case without rotation, so rotation cannot replace waterfilling in general. It nearly can on real layers: rotation cuts the cost of an η-fraction of outlier eigenvalues of size μ from about ημ to η^2μ/2 to leading order, but cannot remove spectral spread on a logarithmic scale; for non-Gaussian weights it also adds a weight-side rate cost, which we characterize exactly under a standard high-rate assumption. On $96$ Hessians of Qwen3 and Llama-3.1 the rotated gap has median $0.04$ bit and maximum $0.36$ bit, against $2−−5$ bits in the worst case for the same condition numbers, and the limit matches it to within $0.009$ bit.
ER -