OSFP4: Joint Optimization of Diagonal Smoothing and Block Scales for NVFP4 Quantization
Abstract
NVFP4 is an attractive datatype for large language model (LLM) inference, offering compact storage and native tensor-core acceleration. However, preserving accuracy using NVFP4 requires careful quantization. In this work we develop a novel quantization scheme called Optimized Smoothing and Scaling for NVFP4 (OSFP4). For each linear projection it uses a diagonal smoothing matrix whose entries are optimized to minimize the squared matrix-product quantization error under NVFP4, taking into account the rounding procedure that is used (either round-to-nearest, or GPTQ-style successive interference cancellation). This requires performing joint optimization on the smoothing entries as well as the block scales, which is facilitated by analyzing a multiplicative-dither FP4 quantizer instead of the fixed deterministic one. Experiments show that OSFP4 achieves the highest average accuracy among the evaluated competitors in the corresponding quantization settings, while retaining approximately 94-97\% of vendor NVFP4 prefill throughput on the measured workloads. Our code is available in https://github.com/neriahbd/OSFP4
Related
- Hereditary lowerability of diagonal entropy scales
- Discrete eigenvalue optimization from entropic smoothing and first-order methods
- Error Bounds and a Smoothing Penalty Method for Binary Orthogonal Optimization
- A Unifying View of Distributionally Robust Optimization for Probability Smoothing via Maximum Entropy
- Adding Heavy-Ball Momentum to Distributed Constrained Optimization for Energy Resource Management Subject to Time-Delays and Quantization
BibTeX
@misc{david2026osfp4,
title = {{OSFP4: Joint Optimization of Diagonal Smoothing and Block Scales for NVFP4 Quantization}},
author = {Neriah Ben David and Ori Meir and Or Ordentlich},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.08231},
eprint = {2610.08231},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.08231},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - OSFP4: Joint Optimization of Diagonal Smoothing and Block Scales for NVFP4 Quantization
AU - Neriah Ben David
AU - Ori Meir
AU - Or Ordentlich
PY - 2026
DA - 2026/10/06
PB - arXiv
DO - 10.48550/arxiv.2610.08231
UR - https://arxiv.org/abs/2610.08231
AB - NVFP4 is an attractive datatype for large language model (LLM) inference, offering compact storage and native tensor-core acceleration. However, preserving accuracy using NVFP4 requires careful quantization. In this work we develop a novel quantization scheme called Optimized Smoothing and Scaling for NVFP4 (OSFP4). For each linear projection it uses a diagonal smoothing matrix whose entries are optimized to minimize the squared matrix-product quantization error under NVFP4, taking into account the rounding procedure that is used (either round-to-nearest, or GPTQ-style successive interference cancellation). This requires performing joint optimization on the smoothing entries as well as the block scales, which is facilitated by analyzing a multiplicative-dither FP4 quantizer instead of the fixed deterministic one. Experiments show that OSFP4 achieves the highest average accuracy among the evaluated competitors in the corresponding quantization settings, while retaining approximately 94-97\% of vendor NVFP4 prefill throughput on the measured workloads. Our code is available in https://github.com/neriahbd/OSFP4
ER -