Dr. OPD: Learning What to Follow for Optimal On-Policy Distillation of Large Language Models
Abstract
On-policy distillation (OPD) trains a student on its own generated responses using dense, token-level supervision from a stronger teacher. Vanilla OPD treats all teacher signals equally, assuming that the teacher's supervision is equally important for every token. However, teacher signals at different tokens may have very different effects on the student's performance: some correct important reasoning errors, while others have little effect on the final answer. Motivated by this observation, we introduce Dr. OPD (OPD Done Right), which defines the optimal weighted OPD to maximize the student's performance. We formulate Dr. OPD as a bilevel optimization problem in which the student learns from weighted teacher supervision, while the weights are selected to maximize the expected reward of the resulting student. To solve Dr. OPD, we develop an efficient iterative solver that updates the token weights and student policy alternatively. At each round, it updates weights in closed form and then takes one gradient step on the resulting weighted OPD objective. Under regularity conditions, we show that this weighted update achieves a higher expected reward than a vanilla OPD update. Empirically, across strong-to-weak and same-size distillation on math and code, Dr. OPD consistently outperforms all evaluated baselines. In particular, in the strong-to-weak distillation setting, Dr. OPD improves average math performance by $9.7$ points over vanilla OPD, and enables the smaller student to surpass its larger teacher.
Related
- When to Follow the Teacher: Advantage-Based On-Policy Distillation
- OTROPE: Optimal Transport-based Robust Off-policy Evaluation for Large Language Models
- OC-OPD: Observation-coordinated Supervision for On-Policy Distillation
- Understanding Off- vs On-Policy Distillation: A Tale of Distinct Training Objectives
- Function-Space Temporal Self-Distillation for Policy Optimization
BibTeX
@misc{wang2026learning,
title = {{Dr. OPD: Learning What to Follow for Optimal On-Policy Distillation of Large Language Models}},
author = {Zhenyu Wang and Tianze Wang and Linjun Zhang and Yifan Hu},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2609.38025},
eprint = {2609.38025},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2609.38025},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Dr. OPD: Learning What to Follow for Optimal On-Policy Distillation of Large Language Models
AU - Zhenyu Wang
AU - Tianze Wang
AU - Linjun Zhang
AU - Yifan Hu
PY - 2026
DA - 2026/09/29
PB - arXiv
DO - 10.48550/arxiv.2609.38025
UR - https://arxiv.org/abs/2609.38025
AB - On-policy distillation (OPD) trains a student on its own generated responses using dense, token-level supervision from a stronger teacher. Vanilla OPD treats all teacher signals equally, assuming that the teacher's supervision is equally important for every token. However, teacher signals at different tokens may have very different effects on the student's performance: some correct important reasoning errors, while others have little effect on the final answer. Motivated by this observation, we introduce Dr. OPD (OPD Done Right), which defines the optimal weighted OPD to maximize the student's performance. We formulate Dr. OPD as a bilevel optimization problem in which the student learns from weighted teacher supervision, while the weights are selected to maximize the expected reward of the resulting student. To solve Dr. OPD, we develop an efficient iterative solver that updates the token weights and student policy alternatively. At each round, it updates weights in closed form and then takes one gradient step on the resulting weighted OPD objective. Under regularity conditions, we show that this weighted update achieves a higher expected reward than a vanilla OPD update. Empirically, across strong-to-weak and same-size distillation on math and code, Dr. OPD consistently outperforms all evaluated baselines. In particular, in the strong-to-weak distillation setting, Dr. OPD improves average math performance by $9.7$ points over vanilla OPD, and enables the smaller student to surpass its larger teacher.
ER -