SR-OPSD: Self-Referenced On-Policy Self-Distillation
Abstract
On-policy self-distillation (OPSD) converts feedback into dense token-level supervision on student-generated trajectories, complementing reinforcement learning with sparse outcome rewards. Its self-teacher, derived from the student's current or exponentially averaged parameters and conditioned on additional context, evolves alongside the student and its rollout context distribution. The benefit of modifying this moving target depends on how target--student probability mismatches translate into updates. We propose \emph{Self-Referenced On-Policy Self-Distillation (SR-OPSD)}, which constructs a normalized geometric target from the self-teacher and a frozen initial policy, then minimizes the forward Rényi divergence from this target to the student. The interpolation coefficient controls the self-teacher's contribution, while the Rényi order controls the power weighting of target-to-student probability ratios in the gradient. For fixed contexts and target components, we establish a conditional variational characterization and derive the exact token-logit gradient, revealing how anchoring and projection jointly shape the effective update target. Experiments across scientific reasoning, tool use, mathematical reasoning, and code generation demonstrate strong performance across multiple model families and scales. Ablations further show that reference anchoring can improve or degrade performance depending on the projection objective, supporting the joint design of target construction and projection geometry.
Related
- Function-Space Temporal Self-Distillation for Policy Optimization
- OC-OPD: Observation-coordinated Supervision for On-Policy Distillation
- Understanding Off- vs On-Policy Distillation: A Tale of Distinct Training Objectives
- When to Follow the Teacher: Advantage-Based On-Policy Distillation
- Counterfactual Constraint-Conditioned On-Policy Distillation for Multi-Constraint Instruction Following
BibTeX
@misc{sun2026sropsd,
title = {{SR-OPSD: Self-Referenced On-Policy Self-Distillation}},
author = {Zhuo Sun and Entong Li and Yanlong Zhao and Xiaoyuan Cheng and Wenxuan Yuan and Kaiyu Li and Che Liu and Huihang Liu and Baihua He and Xinyu Zhang and Harrison Bo Hua Zhu and Li Zeng},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2608.09745},
eprint = {2608.09745},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2608.09745},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - SR-OPSD: Self-Referenced On-Policy Self-Distillation
AU - Zhuo Sun
AU - Entong Li
AU - Yanlong Zhao
AU - Xiaoyuan Cheng
AU - Wenxuan Yuan
AU - Kaiyu Li
AU - Che Liu
AU - Huihang Liu
AU - Baihua He
AU - Xinyu Zhang
AU - Harrison Bo Hua Zhu
AU - Li Zeng
PY - 2026
DA - 2026/09/29
PB - arXiv
DO - 10.48550/arxiv.2608.09745
UR - https://arxiv.org/abs/2608.09745
AB - On-policy self-distillation (OPSD) converts feedback into dense token-level supervision on student-generated trajectories, complementing reinforcement learning with sparse outcome rewards. Its self-teacher, derived from the student's current or exponentially averaged parameters and conditioned on additional context, evolves alongside the student and its rollout context distribution. The benefit of modifying this moving target depends on how target--student probability mismatches translate into updates. We propose \emph{Self-Referenced On-Policy Self-Distillation (SR-OPSD)}, which constructs a normalized geometric target from the self-teacher and a frozen initial policy, then minimizes the forward Rényi divergence from this target to the student. The interpolation coefficient controls the self-teacher's contribution, while the Rényi order controls the power weighting of target-to-student probability ratios in the gradient. For fixed contexts and target components, we establish a conditional variational characterization and derive the exact token-logit gradient, revealing how anchoring and projection jointly shape the effective update target. Experiments across scientific reasoning, tool use, mathematical reasoning, and code generation demonstrate strong performance across multiple model families and scales. Ablations further show that reference anchoring can improve or degrade performance depending on the projection objective, supporting the joint design of target construction and projection geometry.
ER -