Failure Is the Mother of Success: Recovering Supervision with Grounding Hindsight
Abstract
When every rollout for a problem fails, GRPO receives neither a successful exemplar nor a nonzero group-relative advantage, so it learns nothing from exactly the problems it most needs to learn. The failed reasoning can nevertheless suggest what to try next. We present Replay-Selected Hindsight Distillation (RSHD), which turns failed attempts into an additional, tested source of supervision during GRPO training. The policy writes a corrective note about each failed attempt without seeing a reference answer or solution, then re-solves the problem with the note in context. A note is admitted only if its replays succeed more often than the original group did, after screening for notes that state the answer, and an admitted replay is distilled into the plain-prompt policy. Because admission is decided by fresh successes rather than by the failed rollout itself, the procedure can recover supervision from groups in which every attempt failed, and it applies equally to failures in mixed groups. Only the policy is trained, the only judge is the outcome verifier, and deployment requires no notes or extra inference. Across model families and sizes and two training domains, RSHD consistently improves over GRPO on in-domain benchmarks, and the gains carry over to out-of-domain evaluation sets.
Related
- Finite-Time Tracking Success or Failure in the Pursuer--Evader Model
- The Optimizer's Windfall: Bounding Hindsight Gain under Random Restrictions
- Beyond an Ethos or Inflexible Checklist of Research Practices: Grounding Open Science in a Biomedical Research Ethics Framework
- OrbitTAMP: Grounding Language Models for Task and Motion Planning in Spacecraft Rendezvous
- Recovering curves from isogeny classes of Prym varieties
BibTeX
@misc{yu2026failure,
title = {{Failure Is the Mother of Success: Recovering Supervision with Grounding Hindsight}},
author = {Sheldon Yu and Tong Yu and Nikki Lijing Kuang and Zhendong Chu and Zhenwei Tang and Sungchul Kim and Lina Yao and Julian McAuley and Junda Wu},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.21018},
eprint = {2610.21018},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.21018},
note = {CC BY-NC-SA 4.0},
}
RIS
TY - UNPB
TI - Failure Is the Mother of Success: Recovering Supervision with Grounding Hindsight
AU - Sheldon Yu
AU - Tong Yu
AU - Nikki Lijing Kuang
AU - Zhendong Chu
AU - Zhenwei Tang
AU - Sungchul Kim
AU - Lina Yao
AU - Julian McAuley
AU - Junda Wu
PY - 2026
DA - 2026/10/14
PB - arXiv
DO - 10.48550/arxiv.2610.21018
UR - https://arxiv.org/abs/2610.21018
AB - When every rollout for a problem fails, GRPO receives neither a successful exemplar nor a nonzero group-relative advantage, so it learns nothing from exactly the problems it most needs to learn. The failed reasoning can nevertheless suggest what to try next. We present Replay-Selected Hindsight Distillation (RSHD), which turns failed attempts into an additional, tested source of supervision during GRPO training. The policy writes a corrective note about each failed attempt without seeing a reference answer or solution, then re-solves the problem with the note in context. A note is admitted only if its replays succeed more often than the original group did, after screening for notes that state the answer, and an admitted replay is distilled into the plain-prompt policy. Because admission is decided by fresh successes rather than by the failed rollout itself, the procedure can recover supervision from groups in which every attempt failed, and it applies equally to failures in mixed groups. Only the policy is trained, the only judge is the outcome verifier, and deployment requires no notes or extra inference. Across model families and sizes and two training domains, RSHD consistently improves over GRPO on in-domain benchmarks, and the gains carry over to out-of-domain evaluation sets.
ER -