A Critical Look at Power Sampling when Test Time Sharpening LLMs
Abstract
Reinforcement learning with verifiable rewards (RLVR) has substantially improved LLM reasoning, yet growing evidence suggests it mainly performs distribution sharpening: concentrating probability mass on reasoning paths the base model already produces. Recent work therefore seeks RLVR's gains without needing reward signals, by sampling from the power distribution πα∝ πbaseα via Markov chain Monte Carlo or sequential Monte Carlo. We show that these samplers fail to converge to πα under realistic compute budgets, and that their gains are largely recreated via simple Best-of-N with appropriate grading . This suggests they succeed by selecting high-probability sequences, not by approximating πα, raising a natural question: is the power distribution itself a worthwhile target? We test this hypothesis directly by fine-tuning the base model to match πα, sidestepping sampler convergence issues entirely. Despite approximating the target far more faithfully than existing samplers, this strategy yields gains that are noisy and specific to certain models and sampling tasks. Together, our results suggest that the value of power sampling lies not in faithfully targeting πα, but in the simpler goal of extracting sequences the base model already deems likely.
Related
- Opening Rankings from Engine Self-Play Are Less Certain Than They Look: Sampling Error, Draws, and Strength Dependence in Strength-Limited Stockfish Games
- When n is not enough: Assessing the interplay of spatial sampling error and sample selection in abundance estimation
- When May a Bandit Leave Its Anchor? E-Process-Authorized Thompson Sampling under Non-stationarity
- When can a power series be analytically continued?
- When is multivariate kriging worthwhile? Design geometry, parameter sharing and response coupling under heterotopic sampling
BibTeX
@misc{talwar2026critical,
title = {{A Critical Look at Power Sampling when Test Time Sharpening LLMs}},
author = {Abiel Malkani Talwar and Christopher Nemeth and Henry Moss},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.13653},
eprint = {2610.13653},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.13653},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - A Critical Look at Power Sampling when Test Time Sharpening LLMs
AU - Abiel Malkani Talwar
AU - Christopher Nemeth
AU - Henry Moss
PY - 2026
DA - 2026/10/09
PB - arXiv
DO - 10.48550/arxiv.2610.13653
UR - https://arxiv.org/abs/2610.13653
AB - Reinforcement learning with verifiable rewards (RLVR) has substantially improved LLM reasoning, yet growing evidence suggests it mainly performs distribution sharpening: concentrating probability mass on reasoning paths the base model already produces. Recent work therefore seeks RLVR's gains without needing reward signals, by sampling from the power distribution π_α∝ π_base^α via Markov chain Monte Carlo or sequential Monte Carlo. We show that these samplers fail to converge to π_α under realistic compute budgets, and that their gains are largely recreated via simple Best-of-N with appropriate grading . This suggests they succeed by selecting high-probability sequences, not by approximating π_α, raising a natural question: is the power distribution itself a worthwhile target? We test this hypothesis directly by fine-tuning the base model to match π_α, sidestepping sampler convergence issues entirely. Despite approximating the target far more faithfully than existing samplers, this strategy yields gains that are noisy and specific to certain models and sampling tasks. Together, our results suggest that the value of power sampling lies not in faithfully targeting π_α, but in the simpler goal of extracting sequences the base model already deems likely.
ER -