Deflating the Hessian: Rank-4 W4A4 Quantization for Multimodal Diffusion Transformers

Abstract

In diffusion transformers, low-rank branches can mitigate 4-bit weight--activation (W4A4) post-training quantization (PTQ) loss by decomposing each weight into a low-bit residual and a high-precision low-rank component. Existing low-rank PTQ approaches, however, either optimize low-rank compensation and residual quantization separately, often requiring higher ranks, or rely on second-order weight updates without explicitly modeling activation quantization error, which becomes particularly pronounced under 4-bit quantization. To address these limitations, we present HSVDQuant, a unified framework modeling low-rank-assisted W4A4 PTQ as a coupled calibration problem and deriving optimization-based solvers from the joint objective. Eliminating the output-side low-rank factor yields a deflated Hessian that discounts residual errors already captured by the low-rank component, while an activation-noise surrogate is incorporated to suppress activation quantization error. Across five diffusion backbones, rank-4 HSVDQuant consistently outperforms rank-4 SVDQuant in PSNR and LPIPS. It further surpasses rank-32 SVDQuant on SANA-1.6B, FLUX.1-schnell, and FLUX.1-dev with an 8× smaller rank and up to 6.25× faster quantization. Furthermore, on the Qwen3-8B LLM, rank-4 HSVDQuant improves MMLU accuracy from 61.50% to 68.17% over rank-32 SVDQuant. Overall, HSVDQuant achieves better W4A4 performance with substantially lower rank and quantization cost.

Diffusion Transformers are Provably Optimal In-context Generators
Guoji Fu, Tomoya Wakayama, Ryotaro Kawata +3
arXiv4 Oct 2026PreprintCC BY 4.0Full textStatisticsarXiv
Transformers as In-Context Samplers: From Closed-Form Diffusion to Estimation-Free Sampling
Arman Adibi, Alireza Jafari, Mohammad Ghavamzadeh +1
arXiv11 Sep 2026PreprintCC BY 4.0Full textStatisticsarXiv
Optimal Weighted L² Hessian Estimates for Parabolic Equations under General Diffusion Marginals
Jihao Long, Zhenhua Zhao
arXiv8 Oct 2026PreprintarXiv licenceFull textAnalysis & PDEs ProbabilityarXiv
Joint Branch-Space Transform Coding for Diffusion Activation Quantization with Classifier-Free Guidance
Mingrun Jiang, Yuejia Liu, Zishan Shao +11
arXiv1 Oct 2026PreprintCC BY 4.0Full textStatisticsarXiv
Towards Digital Halftoning on Closed Manifolds--An Error Diffusion Scheme for the 2D Torus based on Sigma-Delta Quantization along the Rank-one Lattice
Felix Krahmer, Alessandro Lupoli
arXiv15 Sep 2026PreprintarXiv licenceFull textGeneral mathematicsarXiv

Cite

BibTeX RIS

BibTeX

@misc{wang2026deflating,
  title = {{Deflating the Hessian: Rank-4 W4A4 Quantization for Multimodal Diffusion Transformers}},
  author = {Shiwen Wang and Pengxiang Zhao and Xiaoming Yuan},
  howpublished = {arXiv},
  year = {2026},
  month = oct,
  doi = {10.48550/arxiv.2610.11315},
  eprint = {2610.11315},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.11315},
  note = {CC BY 4.0},
}

RIS

TY  - UNPB
TI  - Deflating the Hessian: Rank-4 W4A4 Quantization for Multimodal Diffusion Transformers
AU  - Shiwen Wang
AU  - Pengxiang Zhao
AU  - Xiaoming Yuan
PY  - 2026
DA  - 2026/10/09
PB  - arXiv
DO  - 10.48550/arxiv.2610.11315
UR  - https://arxiv.org/abs/2610.11315
AB  - In diffusion transformers, low-rank branches can mitigate 4-bit weight--activation (W4A4) post-training quantization (PTQ) loss by decomposing each weight into a low-bit residual and a high-precision low-rank component. Existing low-rank PTQ approaches, however, either optimize low-rank compensation and residual quantization separately, often requiring higher ranks, or rely on second-order weight updates without explicitly modeling activation quantization error, which becomes particularly pronounced under 4-bit quantization. To address these limitations, we present HSVDQuant, a unified framework modeling low-rank-assisted W4A4 PTQ as a coupled calibration problem and deriving optimization-based solvers from the joint objective. Eliminating the output-side low-rank factor yields a deflated Hessian that discounts residual errors already captured by the low-rank component, while an activation-noise surrogate is incorporated to suppress activation quantization error. Across five diffusion backbones, rank-4 HSVDQuant consistently outperforms rank-4 SVDQuant in PSNR and LPIPS. It further surpasses rank-32 SVDQuant on SANA-1.6B, FLUX.1-schnell, and FLUX.1-dev with an 8× smaller rank and up to 6.25× faster quantization. Furthermore, on the Qwen3-8B LLM, rank-4 HSVDQuant improves MMLU accuracy from 61.50% to 68.17% over rank-32 SVDQuant. Overall, HSVDQuant achieves better W4A4 performance with substantially lower rank and quantization cost.
ER  -

Abstract

Related

Cite