When to Follow the Teacher: Advantage-Based On-Policy Distillation
Abstract
Online Policy Distillation (OPD) often outperforms offline methods such as Supervised Fine-Tuning (SFT) by training on trajectories sampled directly from the student. However, standard OPD trains the student to follow the teacher at every state of the trajectory, assuming that teacher guidance is always beneficial. We instead distinguish between what the teacher prefers and what actually helps the current student succeed. We introduce Advantage-Based On-Policy Distillation (AOPD), which weights teacher actions by their counterfactual advantage under the student. Teacher actions that improve the student's expected outcome are reinforced, while those that do not are downweighted. Directly computing these advantages requires trying alternative teacher actions at each step and running new rollouts from each. Instead, we show that the AOPD objective can be estimated using only standard student rollouts and their terminal rewards. Our estimator uses teacher importance weights on the student-sampled actions to provide unbiased advantage-weighted supervision, without any additional rollouts or reward evaluations. This allows AOPD to focus teacher supervision on actions that help the current student while preserving the rollout efficiency of standard OPD. Experiments on math, coding, and Medqa show gains over distillation and RL baselines in both external-teacher and self-distillation settings.
Related
- Dr. OPD: Learning What to Follow for Optimal On-Policy Distillation of Large Language Models
- Iterative Calibration and Learning: Mitigating Teacher Bias in LLM Distillation without Target-Domain Reward Feedback
- MUON-RT Reusing the Teacher's Trajectory for Momentum-Free Muon for LLM Self-Distillation
- When AI Becomes a Resource: AI-Supported Teacher Design Capacity in Mathematics Teacher Education
- OC-OPD: Observation-coordinated Supervision for On-Policy Distillation
BibTeX
@misc{sabbaghi2026when,
title = {{When to Follow the Teacher: Advantage-Based On-Policy Distillation}},
author = {Mahdi Sabbaghi and George Pappas and Hamed Hassani and Adel Javanmard},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.31563},
eprint = {2610.31563},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.31563},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - When to Follow the Teacher: Advantage-Based On-Policy Distillation
AU - Mahdi Sabbaghi
AU - George Pappas
AU - Hamed Hassani
AU - Adel Javanmard
PY - 2026
DA - 2026/10/22
PB - arXiv
DO - 10.48550/arxiv.2610.31563
UR - https://arxiv.org/abs/2610.31563
AB - Online Policy Distillation (OPD) often outperforms offline methods such as Supervised Fine-Tuning (SFT) by training on trajectories sampled directly from the student. However, standard OPD trains the student to follow the teacher at every state of the trajectory, assuming that teacher guidance is always beneficial. We instead distinguish between what the teacher prefers and what actually helps the current student succeed. We introduce Advantage-Based On-Policy Distillation (AOPD), which weights teacher actions by their counterfactual advantage under the student. Teacher actions that improve the student's expected outcome are reinforced, while those that do not are downweighted. Directly computing these advantages requires trying alternative teacher actions at each step and running new rollouts from each. Instead, we show that the AOPD objective can be estimated using only standard student rollouts and their terminal rewards. Our estimator uses teacher importance weights on the student-sampled actions to provide unbiased advantage-weighted supervision, without any additional rollouts or reward evaluations. This allows AOPD to focus teacher supervision on actions that help the current student while preserving the rollout efficiency of standard OPD. Experiments on math, coding, and Medqa show gains over distillation and RL baselines in both external-teacher and self-distillation settings.
ER -