SpectrumQuant: Spectrum Alignment for LLM Quantization

Abstract

Orthogonal transforms have emerged as an effective tool for low-bit quantization of large language models (LLMs), as they reshape activation distributions without changing the underlying model function. Existing approaches typically rely either on fixed data-agnostic transforms, such as Hadamard rotations, or on calibration-time optimization of learned rotations. We study rotation selection from first principles and show that maximizing covariance leads to spectral alignment and provides a principled criterion for constructing quantization-friendly orthogonal transforms. Based on this insight, we propose SpectrumQuant, a closed-form rotation method that estimates activation covariance from calibration data and uses it to construct a covariance-aware orthogonal transform for post-training quantization. Our formulation characterizes quantization-friendly rotations as compositions of alignment and mixing components, while recovering Hadamard matrices as a special case. Interestingly, a class of butterfly matrices belongs to this category. In the realm of LLM quantization, SpectrumQuant bridges the gap between data-agnostic structured rotations and learned rotations, providing an effective and efficient data-aware alternative for post-training quantization. Experiments on Llama and Mistral models show that SpectrumQuant is competitive with state-of-the-art rotation strategies while requiring less than one-third of the GPU memory and significantly less time than learned rotations approaches like SpinQuant and DartQuant.

Discrete Mixed Quantization for Cat Maps
Elena Kim, Qiaochu Ma
arXiv29 Oct 2026PreprintarXiv licenceFull textAnalysis & PDEs Mathematical physicsarXiv
Fuzzy Filtering for Networked Nonlinear Systems with Dynamic Quantization: Quantization-Dependent Event-Triggered Case
Yunbo Guo, Kezheng Cheng, Zengliang Zhang +1
Mathematics31 Oct 2026Peer-reviewedCC BY 4.0General mathematicsCrossref
Calabi-Yau Deformation Quantization
Jakob Ulmer
arXiv25 Sep 2026PreprintCC BY-NC-SA 4.0Full textAlgebra & number theory Geometry & topologyarXiv
Exponential Green energies, quantization, and minimizers on compact Riemann surfaces
Haakan Hedenmalm, Song-Yan Xie
arXiv10 Oct 2026PreprintarXiv licenceFull textAnalysis & PDEs Geometry & topologyarXiv
A note on the BRST-BV Formalism Explanation of Deformation Quantization and Algebraic Index Theorem
Lihao Zhao
arXiv3 Nov 2026PreprintCC BY 4.0Full textAlgebra & number theory Geometry & topologyarXiv

Cite

BibTeX RIS

BibTeX

@misc{haldar2026spectrumquant,
  title = {{SpectrumQuant: Spectrum Alignment for LLM Quantization}},
  author = {Dwaipayan Haldar and Indranil Patra and Azhar Yousuf and Chandra Sekhar Seelamantula},
  howpublished = {arXiv},
  year = {2026},
  month = oct,
  doi = {10.48550/arxiv.2610.20303},
  eprint = {2610.20303},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.20303},
}

RIS

TY  - UNPB
TI  - SpectrumQuant: Spectrum Alignment for LLM Quantization
AU  - Dwaipayan Haldar
AU  - Indranil Patra
AU  - Azhar Yousuf
AU  - Chandra Sekhar Seelamantula
PY  - 2026
DA  - 2026/10/23
PB  - arXiv
DO  - 10.48550/arxiv.2610.20303
UR  - https://arxiv.org/abs/2610.20303
AB  - Orthogonal transforms have emerged as an effective tool for low-bit quantization of large language models (LLMs), as they reshape activation distributions without changing the underlying model function. Existing approaches typically rely either on fixed data-agnostic transforms, such as Hadamard rotations, or on calibration-time optimization of learned rotations. We study rotation selection from first principles and show that maximizing covariance leads to spectral alignment and provides a principled criterion for constructing quantization-friendly orthogonal transforms. Based on this insight, we propose SpectrumQuant, a closed-form rotation method that estimates activation covariance from calibration data and uses it to construct a covariance-aware orthogonal transform for post-training quantization. Our formulation characterizes quantization-friendly rotations as compositions of alignment and mixing components, while recovering Hadamard matrices as a special case. Interestingly, a class of butterfly matrices belongs to this category. In the realm of LLM quantization, SpectrumQuant bridges the gap between data-agnostic structured rotations and learned rotations, providing an effective and efficient data-aware alternative for post-training quantization. Experiments on Llama and Mistral models show that SpectrumQuant is competitive with state-of-the-art rotation strategies while requiring less than one-third of the GPU memory and significantly less time than learned rotations approaches like SpinQuant and DartQuant.
ER  -

Abstract

Related

Cite