Iterative Calibration and Learning: Mitigating Teacher Bias in LLM Distillation without Target-Domain Reward Feedback
Abstract
Large language model (LLM) distillation can propagate teacher bias under covariate shift when target reward feedback is unavailable. We propose Iterative Calibration and Learning (IterCaL), which alternates source-reward teacher calibration through token-level branching with student learning on target prompts. We prove polynomial convergence in expected average KL to the oracle student, the reference-regularized target-reward maximizer within a potentially misspecified student class; a quantified source-to-target calibration discrepancy gives an explicit additive true-return term. Our separation shows that regularized direct teacher matching retains positive oracle-KL error at its population optimum for every shared reference penalty, even when the teacher is stronger than every student. On the same instance, IterCaL converges to the oracle using source rewards alone.
Related
- Entanglement-of-formation potentials for feedback-assisted classical communication and randomness distillation
- Personalized Federated Learning through Global Knowledge Distillation and Local Head Adaptation
- Defence against the dark arts of distillation: Unsupervised discovery of subliminal learning
- Mitigating Model Collapse through Dreaming Learning
- MAEM: Mitigating Recency Bias via a Replay-Free, Capacity-Controlled Dynamic Mixture of Experts for Class-Incremental Graph Learning
BibTeX
@misc{hu2026iterative,
title = {{Iterative Calibration and Learning: Mitigating Teacher Bias in LLM Distillation without Target-Domain Reward Feedback}},
author = {Haichen Hu and Yuheng Zhang and David Simchi-Levi},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2609.17474},
eprint = {2609.17474},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2609.17474},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Iterative Calibration and Learning: Mitigating Teacher Bias in LLM Distillation without Target-Domain Reward Feedback
AU - Haichen Hu
AU - Yuheng Zhang
AU - David Simchi-Levi
PY - 2026
DA - 2026/10/29
PB - arXiv
DO - 10.48550/arxiv.2609.17474
UR - https://arxiv.org/abs/2609.17474
AB - Large language model (LLM) distillation can propagate teacher bias under covariate shift when target reward feedback is unavailable. We propose Iterative Calibration and Learning (IterCaL), which alternates source-reward teacher calibration through token-level branching with student learning on target prompts. We prove polynomial convergence in expected average KL to the oracle student, the reference-regularized target-reward maximizer within a potentially misspecified student class; a quantified source-to-target calibration discrepancy gives an explicit additive true-return term. Our separation shows that regularized direct teacher matching retains positive oracle-KL error at its population optimum for every shared reference penalty, even when the teacher is stronger than every student. On the same instance, IterCaL converges to the oracle using source rewards alone.
ER -