RLTLDR: Self-improvement by Internalizing Self-generated Feedback
Abstract
The common paradigm of reinforcement learning with verifiable rewards (RLVR) is to let agents make multiple attempts at a task, and optimize towards the successful ones. This becomes problematic in the realms of self-improvement, where tasks are so difficult that the agent has a low or even no chance of success, and where there are no teacher models or example solutions to distill from. In this paper, we introduce RLTLDR. After each failed attempt, we show the policy the verifier outputs and let it write its own feedback, in the form of a single TL;DR insight. The next rollout is conditioned on all previous insights, and we sequentially sample rollouts until a solution is found. Moreover, we enable backpropagation on the in-context insights to internalize a direct task to insight mapping. On challenging tool-calling and coding datasets (filtered to Pass@128=0), standard GRPO training of a Qwen 3.5 9B Thinking policy stays flat at a Pass@1 of 0% to 1%. RLTLDR breaks through this learning barrier, achieving a Pass@1 of 14-31% with insights in context during training and, crucially, 12-13% when no insight is in context at eval time. We identify that the key is the task to insight internalization. To study this further, we reduce our approach to SFTLDR, training only on (task, insight) tuples, without showing or backpropagating on any rollouts. Training on only 4k of these tuples recovers almost the full performance of RLTLDR and classical SFT on full rollouts. This demonstrates a promising compacted training paradigm of the form "on this sort of task, keep this sort of thing in mind", which we hope to inspire future research on.
Related
- When Self-Generated Gradients meet Cell Division: Waves and Speed Selection in a Minimal Model
- Gradient self-improvement for mixed local and nonlocal parabolic equations
- A Self-Generated Local Structural Prior for Resolving Target Merging in Electrical Impedance Tomography (EIT)
- Self-improvement for absolutely Cesáro bounded operators
- Which Self-Improvements Should We Trust? Reliable Self-Improvement When Agents Reuse Their Benchmarks
BibTeX
@misc{kirchhof2026rltldr,
title = {{RLTLDR: Self-improvement by Internalizing Self-generated Feedback}},
author = {Michael Kirchhof and Eleonora Gualdoni and Andrew Szot and Khashayar Gatmiry and Aryo Lotfi and Abbas Kazerouni and Omar Attia and Sanjoy Chowdhury and Alexander Toshev},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2609.37633},
eprint = {2609.37633},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2609.37633},
note = {CC BY-NC-ND 4.0},
}
RIS
TY - UNPB
TI - RLTLDR: Self-improvement by Internalizing Self-generated Feedback
AU - Michael Kirchhof
AU - Eleonora Gualdoni
AU - Andrew Szot
AU - Khashayar Gatmiry
AU - Aryo Lotfi
AU - Abbas Kazerouni
AU - Omar Attia
AU - Sanjoy Chowdhury
AU - Alexander Toshev
PY - 2026
DA - 2026/10/19
PB - arXiv
DO - 10.48550/arxiv.2609.37633
UR - https://arxiv.org/abs/2609.37633
AB - The common paradigm of reinforcement learning with verifiable rewards (RLVR) is to let agents make multiple attempts at a task, and optimize towards the successful ones. This becomes problematic in the realms of self-improvement, where tasks are so difficult that the agent has a low or even no chance of success, and where there are no teacher models or example solutions to distill from. In this paper, we introduce RLTLDR. After each failed attempt, we show the policy the verifier outputs and let it write its own feedback, in the form of a single TL;DR insight. The next rollout is conditioned on all previous insights, and we sequentially sample rollouts until a solution is found. Moreover, we enable backpropagation on the in-context insights to internalize a direct task to insight mapping. On challenging tool-calling and coding datasets (filtered to Pass@128=0), standard GRPO training of a Qwen 3.5 9B Thinking policy stays flat at a Pass@1 of 0% to 1%. RLTLDR breaks through this learning barrier, achieving a Pass@1 of 14-31% with insights in context during training and, crucially, 12-13% when no insight is in context at eval time. We identify that the key is the task to insight internalization. To study this further, we reduce our approach to SFTLDR, training only on (task, insight) tuples, without showing or backpropagating on any rollouts. Training on only 4k of these tuples recovers almost the full performance of RLTLDR and classical SFT on full rollouts. This demonstrates a promising compacted training paradigm of the form "on this sort of task, keep this sort of thing in mind", which we hope to inspire future research on.
ER -