Understanding Off- vs On-Policy Distillation: A Tale of Distinct Training Objectives
Abstract
On-policy distillation (OPD) learns from teacher feedback on student-generated responses and has shown promise in reducing forgetting relative to supervised fine-tuning (SFT). However, its benefits and fragility remain incompletely understood. We study sequential distillation from multiple teachers, where the student minimizes its average divergence from the teachers. Forward Kullback--Leibler (KL) divergence yields a weighted arithmetic mixture, while reverse KL yields a normalized weighted geometric aggregate. We develop algorithms that learn these targets under off-policy and on-policy feedback, respectively, establishing logarithmic regret bounds in the tabular setting and extending the analysis to function approximation. By analyzing these aggregation targets, we identify mechanisms that help explain both the benefits and fragility of OPD. Relative to forward KL, reverse KL can better retain a confident expert's preferences under uninformative feedback, but is more sensitive to teachers that assign very low probabilities to correct responses. Its token-level conditionals also reveal a dependence on continuation distributions that can favor incorrect prefixes over long horizons.
Related
- Different representation learning objectives recover distinct latent structures from the same psychometric data
- OC-OPD: Observation-coordinated Supervision for On-Policy Distillation
- Function-Space Temporal Self-Distillation for Policy Optimization
- When to Follow the Teacher: Advantage-Based On-Policy Distillation
- SR-OPSD: Self-Referenced On-Policy Self-Distillation
BibTeX
@misc{di2026understanding,
title = {{Understanding Off- vs On-Policy Distillation: A Tale of Distinct Training Objectives}},
author = {Qiwei Di and Xuheng Li and Kaixuan Ji and Chenggong Zhang and Heyang Zhao and Quanquan Gu},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2609.38666},
eprint = {2609.38666},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2609.38666},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Understanding Off- vs On-Policy Distillation: A Tale of Distinct Training Objectives
AU - Qiwei Di
AU - Xuheng Li
AU - Kaixuan Ji
AU - Chenggong Zhang
AU - Heyang Zhao
AU - Quanquan Gu
PY - 2026
DA - 2026/09/29
PB - arXiv
DO - 10.48550/arxiv.2609.38666
UR - https://arxiv.org/abs/2609.38666
AB - On-policy distillation (OPD) learns from teacher feedback on student-generated responses and has shown promise in reducing forgetting relative to supervised fine-tuning (SFT). However, its benefits and fragility remain incompletely understood. We study sequential distillation from multiple teachers, where the student minimizes its average divergence from the teachers. Forward Kullback--Leibler (KL) divergence yields a weighted arithmetic mixture, while reverse KL yields a normalized weighted geometric aggregate. We develop algorithms that learn these targets under off-policy and on-policy feedback, respectively, establishing logarithmic regret bounds in the tabular setting and extending the analysis to function approximation. By analyzing these aggregation targets, we identify mechanisms that help explain both the benefits and fragility of OPD. Relative to forward KL, reverse KL can better retain a confident expert's preferences under uninformative feedback, but is more sensitive to teachers that assign very low probabilities to correct responses. Its token-level conditionals also reveal a dependence on continuation distributions that can favor incorrect prefixes over long horizons.
ER -