Unbiased Top-k Estimation for On-Policy Distillation
Abstract
On-policy distillation (OPD) is becoming an important component of large language model (LLM) post-training for transferring the reasoning capability of a strong teacher LLM to a weaker student LLM. OPD trains the student by minimizing the reverse KL divergence between the teacher and the student via rollouts generated by the student's policy. However, estimating the gradient of the reverse KL divergence in OPD remains a challenge. Using only the sampled token from the student-generated rollout is computationally cheap but provides limited distributional supervision, which will degrade accuracy. In addition, using the full vocabulary provides complete distributional supervision but is computationally expensive. Therefore, recent works propose Top-k OPD (TK-OPD) that use selected top-k tokens, which provides richer distributional supervision than sampled-token estimation at substantially lower computational cost than full-vocabulary estimation. Unfortunately, using only the selected top-k tokens induces bias, leading to accuracy degradation, as the probability mass outside the selected top-k tokens is discarded. To address the bias of TK-OPD, we propose Tail-Corrected Top-k On-Policy Distillation (TT-OPD). It preserves the advantages of TK-OPD, including rich distributional supervision and low computational cost, while providing an unbiased estimator of the gradient of the reverse KL divergence. The key insight of TT-OPD is to use not only the selected top-k tokens, but also the sampled token from the student-generated rollout, thereby recovering the discarded probability mass in expectation, avoiding the bias. Experimental results demonstrate that TT-OPD significantly outperforms other tested OPD variants.
Related
- On Unbiased Parameter Estimation and Signal Reconstruction
- Error analysis and optimal sample allocation for randomized unbiased estimation: the single term estimator
- OC-OPD: Observation-coordinated Supervision for On-Policy Distillation
- Understanding Off- vs On-Policy Distillation: A Tale of Distinct Training Objectives
- Function-Space Temporal Self-Distillation for Policy Optimization
BibTeX
@misc{meng2026unbiased,
title = {{Unbiased Top-$k$ Estimation for On-Policy Distillation}},
author = {Linjian Meng and Siyuan Gan and YuHan Li and Xiran Wang and Ziyang Ding and Ditang Gou and Yiming Wu and Zhen Zhao},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2609.34447},
eprint = {2609.34447},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2609.34447},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Unbiased Top-k Estimation for On-Policy Distillation
AU - Linjian Meng
AU - Siyuan Gan
AU - YuHan Li
AU - Xiran Wang
AU - Ziyang Ding
AU - Ditang Gou
AU - Yiming Wu
AU - Zhen Zhao
PY - 2026
DA - 2026/10/01
PB - arXiv
DO - 10.48550/arxiv.2609.34447
UR - https://arxiv.org/abs/2609.34447
AB - On-policy distillation (OPD) is becoming an important component of large language model (LLM) post-training for transferring the reasoning capability of a strong teacher LLM to a weaker student LLM. OPD trains the student by minimizing the reverse KL divergence between the teacher and the student via rollouts generated by the student's policy. However, estimating the gradient of the reverse KL divergence in OPD remains a challenge. Using only the sampled token from the student-generated rollout is computationally cheap but provides limited distributional supervision, which will degrade accuracy. In addition, using the full vocabulary provides complete distributional supervision but is computationally expensive. Therefore, recent works propose Top-k OPD (TK-OPD) that use selected top-k tokens, which provides richer distributional supervision than sampled-token estimation at substantially lower computational cost than full-vocabulary estimation. Unfortunately, using only the selected top-k tokens induces bias, leading to accuracy degradation, as the probability mass outside the selected top-k tokens is discarded. To address the bias of TK-OPD, we propose Tail-Corrected Top-k On-Policy Distillation (TT-OPD). It preserves the advantages of TK-OPD, including rich distributional supervision and low computational cost, while providing an unbiased estimator of the gradient of the reverse KL divergence. The key insight of TT-OPD is to use not only the selected top-k tokens, but also the sampled token from the student-generated rollout, thereby recovering the discarded probability mass in expectation, avoiding the bias. Experimental results demonstrate that TT-OPD significantly outperforms other tested OPD variants.
ER -