When Lower Reconstruction Loss Hurts: Distributionally Robust Refinement for Low-Bit LLM Quantization

Abstract

Weight-only post-training quantization (PTQ) relies heavily on reconstruction loss minimization to preserve model quality at low precision. We show that the weights favored by minimizing this loss need not yield better model performance on new tasks. In fact, we find that lower reconstruction loss can even degrade model performance on the same calibration data. Our analysis further shows that weights with lower reconstruction loss on calibration data can have higher loss than other weights when the distribution of input activations changes. Motivated by these observations and our analysis, we propose Distributionally Robust Quantization (DRQ), a post-hoc refinement process that minimizes worst-case reconstruction loss over a constrained set of input activation distributions. DRQ refines the integer codes representing quantized weights within the existing quantization grid, keeping quantization parameters and inference operators unchanged. Extensive experiments show that DRQ improves models quantized by six representative PTQ methods, including AWQ, GPTQ, and ParoQuant, and delivers gains across both dense and mixture-of-experts large language models. These results establish DRQ as a general post-hoc refinement framework for weight-only PTQ, achieving better downstream performance without adding inference overhead.

Data-driven joint optimization of maintenance and spare parts provisioning: A distributionally robust approach
Weijun Xie, Nagi Gebraeel, Stephen Robinson +1
European Journal of Operational Research7 Oct 2026Peer-reviewedLicence not stated2 citedOptimisation & controlHAL
Conservative neural posterior estimation via distributionally robust training
William Laplante, Yuga Hikida, Charita Dellaporta +2
arXiv4 Nov 2026PreprintCC BY 4.0Full textStatisticsarXiv
Optimality of Affine Policies in Distributionally Robust Linear-Quadratic Control with Temporally Correlated Noise
Jakob Nylöf, Daniel Kuhn, John Lygeros +1
arXiv23 Sep 2026PreprintarXiv licenceFull textOptimisation & controlarXiv
Conformal-DRO: Distributionally Robust Optimization with Conformalized Ambiguity Set
Luhao Zhang, Shixiang Zhu
arXiv10 Sep 2026PreprintarXiv licenceFull textOptimisation & control StatisticsarXiv
Certifying Distributionally Robust Decisions from Projection Observations
Pouya Ahadi, Kamran Paynabar
arXiv25 Oct 2026PreprintCC BY 4.0Full textOptimisation & control StatisticsarXiv

Cite

BibTeX RIS

BibTeX

@misc{zhao2026when,
  title = {{When Lower Reconstruction Loss Hurts: Distributionally Robust Refinement for Low-Bit LLM Quantization}},
  author = {Yanlong Zhao and Xiaoyuan Cheng and Huihang Liu and Baihua He and Xinyu Zhang and Harrison Bo Hua Zhu and Wenlong Chen and Li Zeng and Zhuo Sun},
  howpublished = {arXiv},
  year = {2026},
  month = oct,
  doi = {10.48550/arxiv.2610.11226},
  eprint = {2610.11226},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.11226},
  note = {CC BY 4.0},
}

RIS

TY  - UNPB
TI  - When Lower Reconstruction Loss Hurts: Distributionally Robust Refinement for Low-Bit LLM Quantization
AU  - Yanlong Zhao
AU  - Xiaoyuan Cheng
AU  - Huihang Liu
AU  - Baihua He
AU  - Xinyu Zhang
AU  - Harrison Bo Hua Zhu
AU  - Wenlong Chen
AU  - Li Zeng
AU  - Zhuo Sun
PY  - 2026
DA  - 2026/10/08
PB  - arXiv
DO  - 10.48550/arxiv.2610.11226
UR  - https://arxiv.org/abs/2610.11226
AB  - Weight-only post-training quantization (PTQ) relies heavily on reconstruction loss minimization to preserve model quality at low precision. We show that the weights favored by minimizing this loss need not yield better model performance on new tasks. In fact, we find that lower reconstruction loss can even degrade model performance on the same calibration data. Our analysis further shows that weights with lower reconstruction loss on calibration data can have higher loss than other weights when the distribution of input activations changes. Motivated by these observations and our analysis, we propose Distributionally Robust Quantization (DRQ), a post-hoc refinement process that minimizes worst-case reconstruction loss over a constrained set of input activation distributions. DRQ refines the integer codes representing quantized weights within the existing quantization grid, keeping quantization parameters and inference operators unchanged. Extensive experiments show that DRQ improves models quantized by six representative PTQ methods, including AWQ, GPTQ, and ParoQuant, and delivers gains across both dense and mixture-of-experts large language models. These results establish DRQ as a general post-hoc refinement framework for weight-only PTQ, achieving better downstream performance without adding inference overhead.
ER  -

Abstract

Related

Cite