How RL Reshapes LLM Reasoning: Transferability, Coverage, and Scaling Laws
Abstract
Recent studies on reinforcement learning (RL) report seemingly conflicting evidence about large language model (LLM) reasoning. Training on mathematics can improve performance in other domains, yet gains in Pass@1 can coincide with lower Pass@N than the base model. This raises a fundamental question: does RL expand an LLM's reasoning boundary, or merely reweight its existing reasoning space? We revisit these phenomena across Qwen and Gemma model families, showing both cross-domain gains and forgetting, while coverage at large sampling budgets increases on some tasks and decreases on others. Detailed analysis of solution traces before and after RL indicates a shift in the reasoning strategies the model employs, motivating a two-stage autoregressive policy model that separates \emph{strategy selection} from problem-specific execution. Within this framework, we prove how RL's implicit bias reshapes strategy preferences, allowing gains on some tasks while suppressing strategies required by others. This mechanism can also broaden or narrow coverage at a given sampling budget even without expanding strategy support. We further provide theoretical justifications for log-sigmoid and log-linear scaling laws in RL compute, and evaluate their predictive power. Together, these results connect changes in strategy selection to cross-domain transfer, reasoning coverage, and compute scaling.
Related
- Derivation of the Wave Kinetic Equation: Full Range of Scaling Laws
- Dropout Universality: Scaling Laws and Optimal Scheduling at the Edge-of-Chaos
- Energy scaling laws for thin elastic sheets with topological defects
- Scaling laws for linear-elastic two-variant martensitic phase transitions in general domains
- Scaling laws and energy dissipation in the dynamics of SLE curve as a 1D turbulence model
BibTeX
@misc{cheng2026reshapes,
title = {{How RL Reshapes LLM Reasoning: Transferability, Coverage, and Scaling Laws}},
author = {Ziheng Cheng and Yixiao Huang and Hanlin Zhu and Somayeh Sojoudi},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.04158},
eprint = {2610.04158},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.04158},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - How RL Reshapes LLM Reasoning: Transferability, Coverage, and Scaling Laws
AU - Ziheng Cheng
AU - Yixiao Huang
AU - Hanlin Zhu
AU - Somayeh Sojoudi
PY - 2026
DA - 2026/10/03
PB - arXiv
DO - 10.48550/arxiv.2610.04158
UR - https://arxiv.org/abs/2610.04158
AB - Recent studies on reinforcement learning (RL) report seemingly conflicting evidence about large language model (LLM) reasoning. Training on mathematics can improve performance in other domains, yet gains in Pass@1 can coincide with lower Pass@N than the base model. This raises a fundamental question: does RL expand an LLM's reasoning boundary, or merely reweight its existing reasoning space? We revisit these phenomena across Qwen and Gemma model families, showing both cross-domain gains and forgetting, while coverage at large sampling budgets increases on some tasks and decreases on others. Detailed analysis of solution traces before and after RL indicates a shift in the reasoning strategies the model employs, motivating a two-stage autoregressive policy model that separates \emph{strategy selection} from problem-specific execution. Within this framework, we prove how RL's implicit bias reshapes strategy preferences, allowing gains on some tasks while suppressing strategies required by others. This mechanism can also broaden or narrow coverage at a given sampling budget even without expanding strategy support. We further provide theoretical justifications for log-sigmoid and log-linear scaling laws in RL compute, and evaluate their predictive power. Together, these results connect changes in strategy selection to cross-domain transfer, reasoning coverage, and compute scaling.
ER -