OC-OPD: Observation-coordinated Supervision for On-Policy Distillation
Abstract
Reverse and forward Kullback--Leibler (KL) divergences provide complementary supervision in on-policy distillation: reverse KL refines the student's active token rankings, and forward KL allocates mass to teacher-supported candidates. At student-generated prefixes, we form a four-observation matrix by treating the student and teacher as both candidate sources and evaluators. The diagonal self-observations measure uncertainty, while source-centered cross-observations recover the two KL objectives. We establish an exact identity equating the Fisher inverse-metric coupling of their logit gradients with the sum of these contrasts, yielding a common local descent direction for distinct positive distributions. Observation-coordinated on-policy distillation (OC-OPD) uses observations to set preferences, closed-form bargaining to coordinate gradients, and distributional discrepancy to scale the update. Across three heterogeneous student settings (non-thinking generation, thinking generation, and base-model initialization), OC-OPD achieves strong gains in mathematical reasoning and transfers effectively to out-of-distribution scientific question answering and code generation. Our code is publicly available at: https://github.com/RuoranXu/OC-OPD.
Related
- Understanding Off- vs On-Policy Distillation: A Tale of Distinct Training Objectives
- Function-Space Temporal Self-Distillation for Policy Optimization
- When to Follow the Teacher: Advantage-Based On-Policy Distillation
- SR-OPSD: Self-Referenced On-Policy Self-Distillation
- Counterfactual Constraint-Conditioned On-Policy Distillation for Multi-Constraint Instruction Following
BibTeX
@misc{xu2026ocopd,
title = {{OC-OPD: Observation-coordinated Supervision for On-Policy Distillation}},
author = {Ruoran Xu and Yilun Zhao and Ziyu Chen and Arman Cohan},
howpublished = {arXiv},
year = {2026},
month = nov,
doi = {10.48550/arxiv.2611.01965},
eprint = {2611.01965},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2611.01965},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - OC-OPD: Observation-coordinated Supervision for On-Policy Distillation
AU - Ruoran Xu
AU - Yilun Zhao
AU - Ziyu Chen
AU - Arman Cohan
PY - 2026
DA - 2026/11/01
PB - arXiv
DO - 10.48550/arxiv.2611.01965
UR - https://arxiv.org/abs/2611.01965
AB - Reverse and forward Kullback--Leibler (KL) divergences provide complementary supervision in on-policy distillation: reverse KL refines the student's active token rankings, and forward KL allocates mass to teacher-supported candidates. At student-generated prefixes, we form a four-observation matrix by treating the student and teacher as both candidate sources and evaluators. The diagonal self-observations measure uncertainty, while source-centered cross-observations recover the two KL objectives. We establish an exact identity equating the Fisher inverse-metric coupling of their logit gradients with the sum of these contrasts, yielding a common local descent direction for distinct positive distributions. Observation-coordinated on-policy distillation (OC-OPD) uses observations to set preferences, closed-form bargaining to coordinate gradients, and distributional discrepancy to scale the update. Across three heterogeneous student settings (non-thinking generation, thinking generation, and base-model initialization), OC-OPD achieves strong gains in mathematical reasoning and transfers effectively to out-of-distribution scientific question answering and code generation. Our code is publicly available at: https://github.com/RuoranXu/OC-OPD.
ER -