arXiv:2606. 08432v1 Announce Type: new Abstract: On-policy distillation (OPD) has become a central post-training tool for large language models (LLMs), providing dense per-token teacher supervision along the student's own rollouts.
By Li Jiang, Haoran Xu, Yichuan Ding, Amy Zhang
arXiv:2610.19145v1 Announce Type: new
Abstract: On-policy distillation (OPD) is increasingly studied as a cheaper alternative to reinforcement learning, yet it remains a form of knowledge distillatio...
By Yiping Li, Fan Zhao, Zhibo Hou, Zhiyu An, Wan Du
On-policy distillation (OPD) aligns a student model with a teacher's logit distribution on student-generated trajectories. This approach has achieved strong empirical gains and can often surpass conve...
On-policy distillation (OPD) is increasingly studied as a cheaper alternative to reinforcement learning, yet it remains a form of knowledge distillation (KD) that relies on teacher supervision. We rev...
The paper introduces a method for offline on‑policy distillation that addresses the problem of imperfect teacher supervision. By training on teacher‑successful problems and measuring changes in token likelihoods on teacher‑failed trajectories, the authors derive a learnability signal that weights the distillation loss. This approach improves performance on mathematical reasoning and code generation tasks while reducing computational cost compared to online distillation.
By Yihao Ai, Weilong Yan
The paper proposes Mixture Proposal On-Policy Distillation (MPOPD) to address a mismatch issue in on‑policy distillation (OPD) where the rollout policy is fixed to the current student. By treating the rollout policy as an optimization target, MPOPD derives a geometric mixture of student and teacher policies and introduces a proposal policy for efficient rollout generation. Experiments on 16 reasoning benchmarks show consistent gains over OPD, notably a 6.91% accuracy improvement on math reasoning with Qwen3-4B-Base and a 41% reduction in generation length.
By Sirui Chen, Yunzhe Qi, Haobo Xu, Jiaru Zou, Mengting Ai, Zihao Li, Jingrui He