The paper investigates the statistical properties of Reverse‑Kullback‑Leibler (KL) policy distillation, focusing on how different feedback settings affect sample complexity. It shows that when the teacher reveals its full action distribution, online distillation achieves minimax rates, whereas revealing only the probability of the student‑generated action requires substantially more data. In contrast, offline distillation attains the same minimax rate regardless of whether the teacher’s full distribution or only the probability of the teacher‑generated action is revealed, because teacher‑generated actions expose the teacher’s preferences directly.
By Haolin Liu, Chen-Yu Wei, Julian Zimmert
Offline on-policy distillation gains efficiency by collecting student trajectories and teacher supervision once and reusing them throughout optimization. The same reuse makes imperfect supervision per...
The paper introduces a method for offline on‑policy distillation that addresses the problem of imperfect teacher supervision. By training on teacher‑successful problems and measuring changes in token likelihoods on teacher‑failed trajectories, the authors derive a learnability signal that weights the distillation loss. This approach improves performance on mathematical reasoning and code generation tasks while reducing computational cost compared to online distillation.
By Yihao Ai, Weilong Yan
The paper investigates how the sampled-token reverse-KL loss in on‑policy distillation distributes updates across tokens. By analyzing the gradient of the per‑token K2 estimator, the authors find that tokens with low student probability and large teacher‑student gaps receive disproportionately large gradient norms. They propose Surprise‑aware Reweighting (SuRe), a lightweight weighting rule that further amplifies this allocation, and demonstrate that SuRe improves math metrics on Qwen3 student models without harming out‑of‑domain performance.
By Bing Shao, Jiazheng Zhang, Long Ma, Yujiong Shen, Senjie Jin, Xin Guo, Yuming Yang, Mingxu Chai, Zhiheng Xi, Tao Gui, Qi Zhang, Xuanjing Huang
The paper investigates on‑policy distillation (OPD) versus supervised fine‑tuning (SFT), focusing on how students learn from multiple teachers by minimizing divergence. It shows that using forward KL divergence leads to a weighted arithmetic mixture, while reverse KL produces a normalized weighted geometric aggregate. The authors develop algorithms for both off‑policy and on‑policy settings, prove logarithmic regret bounds in tabular cases, extend the analysis to function approximation, and analyze how these aggregation targets explain OPD’s benefits and fragility.
By Qiwei Di, Xuheng Li, Kaixuan Ji, Chenggong Zhang, Heyang Zhao, Quanquan Gu
The paper investigates whether on‑policy distillation (OPD) is always advantageous for transferring knowledge from a teacher to a student model. It introduces Semi‑OPD, which uses offline rollouts from the initial student, and demonstrates that across 17 teacher‑student pairs (1.5B–235B parameters) Semi‑OPD outperforms OPD in 14 cases, achieving up to +13.6% accuracy and an 11.4× speedup. The authors show that the benefit of OPD depends on the alignment between teacher and student, measured by an output‑token overlap ratio, and argue that effective distillation requires on‑policy behavior relative to both models.
By Siyan Zhao, Yonggan Fu, Jindong Jiang, Shih-Yang Liu, Song Bian, Byung-Kwan Lee, Sharath Turuvekere Sreenivas, Wenliang Dai, Hanrong Ye, Aditya Grover, Pavlo Molchanov