Function-Space Temporal Self-Distillation for Policy Optimization
Abstract
Reinforcement Learning with Verifiable Rewards (RLVR) improves reasoning in language models, but scalar outcomes provide coarse supervision over response trajectories. Self-distillation offers denser token-level guidance by comparing an unassisted policy with a feedback-informed self-teacher, yet the teacher evolves with the student, creating unstable targets and motivating additional regularization. We introduce Function-Space Policy Optimization (FSPO), a single-policy framework that combines current privileged adaptation with function-space temporal consolidation. FSPO stores sparse historical next-token distributions as fixed temporal targets, providing temporal separation without maintaining a persistent teacher model. It further introduces privileged disagreement correction to explicitly suppress locally overvalued actions, reliability-adaptive grounding that strengthens reference supervision when self-generated evidence repeatedly fails, and quality and coverage-aware temporal memory that retains verified behaviors and improves privileged demonstrations. Across reasoning and tool-use benchmarks, FSPO achieves the highest average accuracy under a 5-hour training budget, improving over the strongest baseline by approximately 4% points on average while reducing peak memory from about 70 GB to 56 GB on Olmo3-7B-Instruct.
Related
- OC-OPD: Observation-coordinated Supervision for On-Policy Distillation
- Understanding Off- vs On-Policy Distillation: A Tale of Distinct Training Objectives
- When to Follow the Teacher: Advantage-Based On-Policy Distillation
- SR-OPSD: Self-Referenced On-Policy Self-Distillation
- Counterfactual Constraint-Conditioned On-Policy Distillation for Multi-Constraint Instruction Following
BibTeX
@misc{das2026functionspace,
title = {{Function-Space Temporal Self-Distillation for Policy Optimization}},
author = {Aryan Das and Arnab Bhattacharya and Vinay Kumar Verma},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.19687},
eprint = {2610.19687},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.19687},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Function-Space Temporal Self-Distillation for Policy Optimization
AU - Aryan Das
AU - Arnab Bhattacharya
AU - Vinay Kumar Verma
PY - 2026
DA - 2026/10/14
PB - arXiv
DO - 10.48550/arxiv.2610.19687
UR - https://arxiv.org/abs/2610.19687
AB - Reinforcement Learning with Verifiable Rewards (RLVR) improves reasoning in language models, but scalar outcomes provide coarse supervision over response trajectories. Self-distillation offers denser token-level guidance by comparing an unassisted policy with a feedback-informed self-teacher, yet the teacher evolves with the student, creating unstable targets and motivating additional regularization. We introduce Function-Space Policy Optimization (FSPO), a single-policy framework that combines current privileged adaptation with function-space temporal consolidation. FSPO stores sparse historical next-token distributions as fixed temporal targets, providing temporal separation without maintaining a persistent teacher model. It further introduces privileged disagreement correction to explicitly suppress locally overvalued actions, reliability-adaptive grounding that strengthens reference supervision when self-generated evidence repeatedly fails, and quality and coverage-aware temporal memory that retains verified behaviors and improves privileged demonstrations. Across reasoning and tool-use benchmarks, FSPO achieves the highest average accuracy under a 5-hour training budget, improving over the strongest baseline by approximately 4% points on average while reducing peak memory from about 70 GB to 56 GB on Olmo3-7B-Instruct.
ER -