miniTE: Diagnosing and Bridging the Execution Gap in Native NVFP4 Reinforcement Learning
Abstract
Native NVFP4 promises to accelerate reinforcement learning (RL) for large language models (LLMs), but achieving stable learning across separate training and sampling engines remains challenging. We show that matching model weights and numerical formats is insufficient to ensure consistent policy evaluation, highlighting the importance of cross-engine numerical execution. We introduce miniTE, a lightweight framework that combines execution diagnostics, cross-engine numerical alignment, and efficient native W4A4 (4-bit weights and activations) GEMM computation for LLM RL. It improves numerical execution consistency without modifying the RL objective, while reducing implementation overhead to realize the benefits of native low-bit computation. Long-horizon RL experiments on Qwen3-30B-A3B demonstrate stable training and benchmark performance close to BF16, outperforming naive Transformer Engine (TE) NVFP4 in evaluation quality. On single NVIDIA B300 GPU node, the optimized learner reaches 33,540 tokens/s, yielding 1.39x and 1.47x the throughput of BF16 and TE NVFP4, respectively, for complete learner updates. We released the framework, kernels, diagnostics, and training configurations to make native low-bit RL easier.
Related
- Reinforcement learning on the EPRI AI-ccelerating Unit Commitment benchmark: diagnosing over-commitment and guiding MILP with learned policies
- PhiBE-Q-Learning: Bridging Off-Policy Reinforcement Learning and Continuous-Time Control
- Reinforcement Learning Under Runtime Confounding: Policy Learning and Off-Policy Evaluation
- Deep Reinforcement Learning for Blind Passive Beamforming in RIS-Assisted MIMO Systems
- Toward Maximizing Algebraic Connectivity via Density-Dependent Backbone Construction and Reinforcement Learning Completion
BibTeX
@misc{li2026minite,
title = {{miniTE: Diagnosing and Bridging the Execution Gap in Native NVFP4 Reinforcement Learning}},
author = {Zhen Li and Mingfa Feng and Yiming Zhang and Yang Yu and Changhai Zhou and Pengxiang Li and Dongkuan Wu and Congkai Xie and Hongxia Yang},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.23928},
eprint = {2610.23928},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.23928},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - miniTE: Diagnosing and Bridging the Execution Gap in Native NVFP4 Reinforcement Learning
AU - Zhen Li
AU - Mingfa Feng
AU - Yiming Zhang
AU - Yang Yu
AU - Changhai Zhou
AU - Pengxiang Li
AU - Dongkuan Wu
AU - Congkai Xie
AU - Hongxia Yang
PY - 2026
DA - 2026/10/07
PB - arXiv
DO - 10.48550/arxiv.2610.23928
UR - https://arxiv.org/abs/2610.23928
AB - Native NVFP4 promises to accelerate reinforcement learning (RL) for large language models (LLMs), but achieving stable learning across separate training and sampling engines remains challenging. We show that matching model weights and numerical formats is insufficient to ensure consistent policy evaluation, highlighting the importance of cross-engine numerical execution. We introduce miniTE, a lightweight framework that combines execution diagnostics, cross-engine numerical alignment, and efficient native W4A4 (4-bit weights and activations) GEMM computation for LLM RL. It improves numerical execution consistency without modifying the RL objective, while reducing implementation overhead to realize the benefits of native low-bit computation. Long-horizon RL experiments on Qwen3-30B-A3B demonstrate stable training and benchmark performance close to BF16, outperforming naive Transformer Engine (TE) NVFP4 in evaluation quality. On single NVIDIA B300 GPU node, the optimized learner reaches 33,540 tokens/s, yielding 1.39x and 1.47x the throughput of BF16 and TE NVFP4, respectively, for complete learner updates. We released the framework, kernels, diagnostics, and training configurations to make native low-bit RL easier.
ER -