Multi-Bitwidth Quantization for LLMs Using Additive Codebooks
Abstract
As large language models (LLMs) are increasingly deployed across heterogeneous hardware with varying resource constraints, the ability to adaptively manage the performance-efficiency trade-off without retraining is critical. We propose Drop-by-Drop, a novel multi-bitwidth post-training quantization framework enabling inference-time precision control over LLM weights from a single quantized model. As theoretical motivation, we prove that Gaussian sources are successively refinable under a weighted mean squared error distortion motivated by LLM loss functions: successive refinement across distortion levels incurs no rate penalty relative to encoding at each level separately. Drop-by-Drop approximates this hierarchical structure in practice by incorporating Matryoshka-style supervision into additive codebook training, inducing an ordering in which codebook prefixes yield accurate partial reconstructions at each precision level. Furthermore, a block-Hadamard rotation brings the weight distribution closer to our Gaussian source assumption. The result is a single model that serves multiple bitwidths by dropping codebooks, reducing storage and quantization cost relative to the static per-bitwidth models. Across Qwen, LLaMA, Gemma, and Mistral, Drop-by-Drop achieves lower perplexity than state-of-the-art multi-bitwidth methods with competitive zero-shot accuracy, while our specialized kernels further reduce decoding latency.
Related
- Multi-Domain Clustering via Measure Quantization
- Evaluating the effectiveness of open-weight LLMs for multi-domain Bengali text classification with zero-shot and few-shot prompting
- Derandomizing Dense Binary Hypervector Codebooks for Quantized Scalars
- Perfect Reconstruction Cannot Show Whether Two Codebooks on a VAE's Latent Space Agree
- IrekoGPT: Turning Structured Pruning into Post-Hoc Slimmable LLMs
BibTeX
@misc{babaoglu2026multibitwidth,
title = {{Multi-Bitwidth Quantization for LLMs Using Additive Codebooks}},
author = {Liza Babaoglu and Shuangyi Chen and Ashish Khisti},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2606.12876},
eprint = {2606.12876},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2606.12876},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Multi-Bitwidth Quantization for LLMs Using Additive Codebooks
AU - Liza Babaoglu
AU - Shuangyi Chen
AU - Ashish Khisti
PY - 2026
DA - 2026/09/26
PB - arXiv
DO - 10.48550/arxiv.2606.12876
UR - https://arxiv.org/abs/2606.12876
AB - As large language models (LLMs) are increasingly deployed across heterogeneous hardware with varying resource constraints, the ability to adaptively manage the performance-efficiency trade-off without retraining is critical. We propose Drop-by-Drop, a novel multi-bitwidth post-training quantization framework enabling inference-time precision control over LLM weights from a single quantized model. As theoretical motivation, we prove that Gaussian sources are successively refinable under a weighted mean squared error distortion motivated by LLM loss functions: successive refinement across distortion levels incurs no rate penalty relative to encoding at each level separately. Drop-by-Drop approximates this hierarchical structure in practice by incorporating Matryoshka-style supervision into additive codebook training, inducing an ordering in which codebook prefixes yield accurate partial reconstructions at each precision level. Furthermore, a block-Hadamard rotation brings the weight distribution closer to our Gaussian source assumption. The result is a single model that serves multiple bitwidths by dropping codebooks, reducing storage and quantization cost relative to the static per-bitwidth models. Across Qwen, LLaMA, Gemma, and Mistral, Drop-by-Drop achieves lower perplexity than state-of-the-art multi-bitwidth methods with competitive zero-shot accuracy, while our specialized kernels further reduce decoding latency.
ER -