RLVR is a Kernel, Not a Function: Statistical Inference for pass@k Crossovers
Abstract
Reinforcement learning with verifiable rewards (RLVR) typically improves pass@1 but can fall behind its base model at large sampling budgets k, commonly interpreted as sharpening existing solutions. Because adjacent pass@k estimates reuse generations, pointwise error bars cannot establish a crossover. We formulate paired inference on the RL-minus-base difference curve: a prompt-level multiplier bootstrap yields a one-sided dominance test and simultaneous confidence bands requiring both an early gain and a later loss. Among nine public RLVR checkpoints at 3k-16k token limits, only one narrowly meets this criterion before multiplicity adjustment; asymmetric truncation can create, mask, or reverse crossovers. On 1,060 deduplicated competition problems at 32k tokens, two 1.5B models exhibit certified crossovers around k≈10-$60$, while two 7B models show no supported loss. A paired variance decomposition shows why adding prompts improves power more than deeper sampling. Prompts with identical base success probability diverge after RLVR, with dispersion replicating across independent generation halves and violating Lipschitz slope bounds for reasonable deterministic maps. Modeling RLVR as a Markov kernel, Jensen's inequality shows that conditional dispersion depresses pass@k for k≥2. Once dispersion is modeled, the conditional-mean log-odds slope is below one in every pair: RLVR improves hard prompts on average, while late-k loss comes from a 2%-3% minority of base-solved prompts collapsing onto a repeated modal wrong answer and resisting prefix rescue up to 4,096 tokens. In controlled runs from one base model on 3,000 new problems, all four GRPO arms fall behind near k≈200 and remain behind through k=4096, while SFT stays significantly ahead throughout.
Related
- Semiparametrically efficient semi-supervised statistical inference
- Statistical inference and applications of Kumaraswamy Odd Perks power Lomax distribution
- Statistical inference for low-rank tensors: Heteroskedasticity, subgaussianity, and applications
- An Efficient Monte Carlo Method for Valid Prior-Free Possibilistic Statistical Inference
- Statistical Uniformization of Surfaces by Heat-Kernel Fisher Information
BibTeX
@misc{yang2026rlvr,
title = {{RLVR is a Kernel, Not a Function: Statistical Inference for pass@$k$ Crossovers}},
author = {Chen Yang and Jun Chen},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2609.22547},
eprint = {2609.22547},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2609.22547},
}
RIS
TY - UNPB
TI - RLVR is a Kernel, Not a Function: Statistical Inference for pass@k Crossovers
AU - Chen Yang
AU - Jun Chen
PY - 2026
DA - 2026/09/25
PB - arXiv
DO - 10.48550/arxiv.2609.22547
UR - https://arxiv.org/abs/2609.22547
AB - Reinforcement learning with verifiable rewards (RLVR) typically improves pass@1 but can fall behind its base model at large sampling budgets k, commonly interpreted as sharpening existing solutions. Because adjacent pass@k estimates reuse generations, pointwise error bars cannot establish a crossover. We formulate paired inference on the RL-minus-base difference curve: a prompt-level multiplier bootstrap yields a one-sided dominance test and simultaneous confidence bands requiring both an early gain and a later loss. Among nine public RLVR checkpoints at 3k-16k token limits, only one narrowly meets this criterion before multiplicity adjustment; asymmetric truncation can create, mask, or reverse crossovers. On 1,060 deduplicated competition problems at 32k tokens, two 1.5B models exhibit certified crossovers around k≈10-$60$, while two 7B models show no supported loss. A paired variance decomposition shows why adding prompts improves power more than deeper sampling. Prompts with identical base success probability diverge after RLVR, with dispersion replicating across independent generation halves and violating Lipschitz slope bounds for reasonable deterministic maps. Modeling RLVR as a Markov kernel, Jensen's inequality shows that conditional dispersion depresses pass@k for k≥2. Once dispersion is modeled, the conditional-mean log-odds slope is below one in every pair: RLVR improves hard prompts on average, while late-k loss comes from a 2%-3% minority of base-solved prompts collapsing onto a repeated modal wrong answer and resisting prefix rescue up to 4,096 tokens. In controlled runs from one base model on 3,000 new problems, all four GRPO arms fall behind near k≈200 and remain behind through k=4096, while SFT stays significantly ahead throughout.
ER -