Beyond Expert Rankings: Contrast Calibration for Mixed-Precision MoE Quantization
Abstract
Mixture-of-experts (MoE) language models activate only a few experts per token, yet all expert weights must be stored. Mixed-precision quantization reduces this storage cost by assigning higher bit-widths to sensitive experts and lower bit-widths to resilient ones, but expert rankings alone do not determine how strongly to weight their errors. To address this gap, we propose NatScale, a contrast-calibrated allocation framework for mixed-precision MoE quantization. It uses a model-specific power transformation to calibrate gradient-based expert weights to an information-theoretic contrast target while preserving rankings. We jointly evaluate each expert's gate, up and down configurations to estimate its output error. We then combine these errors and calibrated weights with an existing prior based on weight spectra, variance and precision decay to select configurations under a storage budget. Across three models, NatScale achieves the lowest native-to-quantized KL divergence and the highest top-token agreement in all nine settings against five allocation baselines. It also yields the lowest perplexity in seven settings. These findings highlight the importance of objective-aware precision allocation.
Related
- Beyond Scalar Sensitivity: Activation-Aware Mixed-Precision LLM Quantization with Cross-Layer Refinement
- Divide et Calibra: Multiclass Local Calibration via Vector Quantization
- SemKV: Semantic Mixed-Precision KV Cache Quantization Guided by the Quality Cliff for Long-Context LLM Inference
- Physically Constrained Kinematic Calibration for High-Precision Robotic Needle Guidance
- When Unpaired Sets Support Shared-Corruption Calibration: Moment Geometry and Two-Sample Precision
BibTeX
@misc{xie2026beyond,
title = {{Beyond Expert Rankings: Contrast Calibration for Mixed-Precision MoE Quantization}},
author = {Renjie Xie and Juncheng Yang and Aoting Hu and Yan Liang and Mingxi Zhang and Zheheng Hong and Wei Xu and Yanzhi Chen},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.20353},
eprint = {2610.20353},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.20353},
}
RIS
TY - UNPB
TI - Beyond Expert Rankings: Contrast Calibration for Mixed-Precision MoE Quantization
AU - Renjie Xie
AU - Juncheng Yang
AU - Aoting Hu
AU - Yan Liang
AU - Mingxi Zhang
AU - Zheheng Hong
AU - Wei Xu
AU - Yanzhi Chen
PY - 2026
DA - 2026/10/14
PB - arXiv
DO - 10.48550/arxiv.2610.20353
UR - https://arxiv.org/abs/2610.20353
AB - Mixture-of-experts (MoE) language models activate only a few experts per token, yet all expert weights must be stored. Mixed-precision quantization reduces this storage cost by assigning higher bit-widths to sensitive experts and lower bit-widths to resilient ones, but expert rankings alone do not determine how strongly to weight their errors. To address this gap, we propose NatScale, a contrast-calibrated allocation framework for mixed-precision MoE quantization. It uses a model-specific power transformation to calibrate gradient-based expert weights to an information-theoretic contrast target while preserving rankings. We jointly evaluate each expert's gate, up and down configurations to estimate its output error. We then combine these errors and calibrated weights with an existing prior based on weight spectra, variance and precision decay to select configurations under a storage budget. Across three models, NatScale achieves the lowest native-to-quantized KL divergence and the highest top-token agreement in all nine settings against five allocation baselines. It also yields the lowest perplexity in seven settings. These findings highlight the importance of objective-aware precision allocation.
ER -