When Does Momentum Help in Reinforcement Learning with Verifiable Rewards?
Abstract
Momentum is a classical tool for accelerating convex optimization and a standard component of modern large language model training. Yet recent empirical work has questioned the usefulness of first-moment momentum in reinforcement learning with verifiable rewards (RLVR). We study this question through a bandit formulation of RLVR with bias-corrected exponential moving average (EMA) momentum, where the EMA coefficient β controls how quickly the weights on past gradients decay. When policy gradients are evaluated exactly, we prove that increasing β within a range that prevents overshoot strictly accelerates optimization. When gradients are estimated from sampled responses, a positive expected gain can be obscured by sampling fluctuations. We show that as the learning rate η decreases, the expected gain with fixed β vanishes faster than these fluctuations, whereas increasing β to keep η/(1−β) within the safe regime preserves a nonvanishing gain that eventually dominates them. Bandit simulations quantitatively validate these predictions. RLVR experiments on Qwen3-4B-Base using SGD with EMA and AdamW support our principle of choosing larger coefficients in practical LLM post-training. On Qwen2.5-32B with AdamW, simply increasing β1 improves mean validation accuracy by nearly $2$ percentage points over the conventional choice, demonstrating the principle's effectiveness at a larger model scale.
Related
- Reinforcement Learning for Hierarchical Reasoning Rewards: Minimax-Optimal Rates with Transformers
- Learning to Solve Stochastic Controls with Unknown Drifts and Running Rewards: Theory, Algorithms and Convergence
- When Momentum Helps Polyak--Ruppert Averaging
- Reinforcement Learning Under Runtime Confounding: Policy Learning and Off-Policy Evaluation
- Deep Reinforcement Learning for Blind Passive Beamforming in RIS-Assisted MIMO Systems
BibTeX
@misc{wang2026when,
title = {{When Does Momentum Help in Reinforcement Learning with Verifiable Rewards?}},
author = {Kaifei Wang and Zikang Shan and Liwei Wang and Yinyu Ye and Han Zhong},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.20413},
eprint = {2610.20413},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.20413},
}
RIS
TY - UNPB
TI - When Does Momentum Help in Reinforcement Learning with Verifiable Rewards?
AU - Kaifei Wang
AU - Zikang Shan
AU - Liwei Wang
AU - Yinyu Ye
AU - Han Zhong
PY - 2026
DA - 2026/10/14
PB - arXiv
DO - 10.48550/arxiv.2610.20413
UR - https://arxiv.org/abs/2610.20413
AB - Momentum is a classical tool for accelerating convex optimization and a standard component of modern large language model training. Yet recent empirical work has questioned the usefulness of first-moment momentum in reinforcement learning with verifiable rewards (RLVR). We study this question through a bandit formulation of RLVR with bias-corrected exponential moving average (EMA) momentum, where the EMA coefficient β controls how quickly the weights on past gradients decay. When policy gradients are evaluated exactly, we prove that increasing β within a range that prevents overshoot strictly accelerates optimization. When gradients are estimated from sampled responses, a positive expected gain can be obscured by sampling fluctuations. We show that as the learning rate η decreases, the expected gain with fixed β vanishes faster than these fluctuations, whereas increasing β to keep η/(1−β) within the safe regime preserves a nonvanishing gain that eventually dominates them. Bandit simulations quantitatively validate these predictions. RLVR experiments on Qwen3-4B-Base using SGD with EMA and AdamW support our principle of choosing larger coefficients in practical LLM post-training. On Qwen2.5-32B with AdamW, simply increasing β_1 improves mean validation accuracy by nearly $2$ percentage points over the conventional choice, demonstrating the principle's effectiveness at a larger model scale.
ER -