RLVR landscapes for iterated multiplications can be benign: Insights from spin-glass theory
Abstract
Despite the importance of reinforcement learning with verifiable rewards (RLVR), the extent to which it can learn new reasoning capabilities remains debated. Here we study the optimization landscape of RLVR on algorithmic tasks, such as iterated group and quasigroup multiplication. To this end, we map entropy-regularized RLVR over myopic tabular policies onto an energy-based (spin-glass) model over deterministic policies. This mapping upper-bounds what RLVR can achieve, and lets us rigorously characterize the landscape in this tabular setting. We show, both theoretically and experimentally, that for a wide class of models and tasks with uncorrelated inputs, this landscape is benign, containing no local minima that could trap RLVR training. Rather, the practical difficulty of these tasks appears to stem, at least in part, from issues such as diffusive barriers and gradient-estimation error in traversing the landscape. These are genuine obstacles that can prevent a solution from being found, but they are distinct from the landscape itself being rugged. We show that these obstacles can often be mitigated through the choice of entropy regulator. Consistent with this theory, we find that a transformer trained from scratch, using only last-token rewards, successfully learns an algorithmic chain of thought for iterated non-Abelian group multiplications.
Related
- Benign population landscapes do not transfer to empirical risk minimization in nonsmooth optimization
- Benign Loss Landscapes Can Coexist with Worst-Case Hardness
- Stable Recovery and Benign Overparameterized Landscapes for Phase Retrieval from Coded Diffraction Patterns
- Using the Renormalized Energy from Ginzburg-Landau Theory to Compute Solution Landscapes in the Reduced Landau-de Gennes Theory
- Weak central polynomials for algebras of multiplications of simple algebras
BibTeX
@misc{rubin2026rlvr,
title = {{RLVR landscapes for iterated multiplications can be benign: Insights from spin-glass theory}},
author = {Noa Rubin and Zohar Ringel},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2609.28625},
eprint = {2609.28625},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2609.28625},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - RLVR landscapes for iterated multiplications can be benign: Insights from spin-glass theory
AU - Noa Rubin
AU - Zohar Ringel
PY - 2026
DA - 2026/09/23
PB - arXiv
DO - 10.48550/arxiv.2609.28625
UR - https://arxiv.org/abs/2609.28625
AB - Despite the importance of reinforcement learning with verifiable rewards (RLVR), the extent to which it can learn new reasoning capabilities remains debated. Here we study the optimization landscape of RLVR on algorithmic tasks, such as iterated group and quasigroup multiplication. To this end, we map entropy-regularized RLVR over myopic tabular policies onto an energy-based (spin-glass) model over deterministic policies. This mapping upper-bounds what RLVR can achieve, and lets us rigorously characterize the landscape in this tabular setting. We show, both theoretically and experimentally, that for a wide class of models and tasks with uncorrelated inputs, this landscape is benign, containing no local minima that could trap RLVR training. Rather, the practical difficulty of these tasks appears to stem, at least in part, from issues such as diffusive barriers and gradient-estimation error in traversing the landscape. These are genuine obstacles that can prevent a solution from being found, but they are distinct from the landscape itself being rugged. We show that these obstacles can often be mitigated through the choice of entropy regulator. Consistent with this theory, we find that a transformer trained from scratch, using only last-token rewards, successfully learns an algorithmic chain of thought for iterated non-Abelian group multiplications.
ER -