How to post-train on a surrogate: Envelope sampling mitigates reward hacking
Abstract
Large language models (LLMs) are commonly post-trained against LLM judges and other cheap surrogates because the true reward, such as human preference, is too expensive to query at scale. This practice often leads to reward hacking, where reinforcement learning against a miscalibrated surrogate leads to undesirable side effects. In this work, we study a setting in which a small number n of model outputs are annotated with ground-truth labels (e.g., from expert review) and used to recalibrate the LLM judge before optimizing against it. Prior approaches to judge recalibration are costly or heuristic, and it is known that on-policy sampling fails when the surrogate is miscalibrated on a rare set of outputs. In this work, we propose envelope sampling, a theoretically-grounded method for judge recalibration that seeks to minimize an upper bound on the regret of the post-trained model under the assumption that the human reward and re-calibrated reward lie in an L2 ball around the judge. We give practical algorithms to sample from the envelope by rejection or by fine-tuning against a modified reward, and experiments on clinical note generation and on a controlled sycophancy task show that recalibrating on envelope samples mitigates reward hacking where recalibrating on base-model samples does not.
Related
- The Surrogate Is Not the Reward: Post-Surrogate Primary-Outcome Acquisition in Contextual Bandits
- Interacting particle guidance for sampling reward-tilted generative priors
- On the nineteenth-century origins of significance testing and p-hacking
- Non-asymptotic Convergence of Average-reward Q-learning with Options
- On optimal stopping problems with full information and rank-dependent reward
BibTeX
@misc{dandapanthula2026posttrain,
title = {{How to post-train on a surrogate: Envelope sampling mitigates reward hacking}},
author = {Sanjit Dandapanthula and Shuvom Sadhuka and Samir Khan and Michael Oberst and Aaditya Ramdas and Alexandra Chouldechova},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.11281},
eprint = {2610.11281},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.11281},
}
RIS
TY - UNPB
TI - How to post-train on a surrogate: Envelope sampling mitigates reward hacking
AU - Sanjit Dandapanthula
AU - Shuvom Sadhuka
AU - Samir Khan
AU - Michael Oberst
AU - Aaditya Ramdas
AU - Alexandra Chouldechova
PY - 2026
DA - 2026/10/08
PB - arXiv
DO - 10.48550/arxiv.2610.11281
UR - https://arxiv.org/abs/2610.11281
AB - Large language models (LLMs) are commonly post-trained against LLM judges and other cheap surrogates because the true reward, such as human preference, is too expensive to query at scale. This practice often leads to reward hacking, where reinforcement learning against a miscalibrated surrogate leads to undesirable side effects. In this work, we study a setting in which a small number n of model outputs are annotated with ground-truth labels (e.g., from expert review) and used to recalibrate the LLM judge before optimizing against it. Prior approaches to judge recalibration are costly or heuristic, and it is known that on-policy sampling fails when the surrogate is miscalibrated on a rare set of outputs. In this work, we propose envelope sampling, a theoretically-grounded method for judge recalibration that seeks to minimize an upper bound on the regret of the post-trained model under the assumption that the human reward and re-calibrated reward lie in an L^2 ball around the judge. We give practical algorithms to sample from the envelope by rejection or by fine-tuning against a modified reward, and experiments on clinical note generation and on a controlled sycophancy task show that recalibrating on envelope samples mitigates reward hacking where recalibrating on base-model samples does not.
ER -