arXiv:2606. 08432v1 Announce Type: new Abstract: On-policy distillation (OPD) has become a central post-training tool for large language models (LLMs), providing dense per-token teacher supervision along the student's own rollouts.
By Li Jiang, Haoran Xu, Yichuan Ding, Amy Zhang
On-policy distillation (OPD) aligns a student model with a teacher's logit distribution on student-generated trajectories. This approach has achieved strong empirical gains and can often surpass conve...
arXiv:2606. 09456v1 Announce Type: new Abstract: On-Policy Distillation (OPD) has become a core technique in the post-training of Large Language Models (LLMs) for transferring knowledge from domain experts to student models.
By Yifan Niu, Han Xiao, Dongyi Liu, Zelong Wang, Dihong Gong, Yasheng Wang, Jia Li
arXiv:2608. 16333v1 Announce Type: cross Abstract: On-policy distillation (OPD) aligns a student model with a teacher's logit distribution on student-generated trajectories.
By Changhui Sun, Lanbo Liu, Hang Lei, Tong Ling, Jiahang Xie, Zhiyong Zheng, Yujia Wang, Hao Liu, Feng Xiao, Lu Liu, Yanlong Du, Zifeng Cheng, Ziwei Jiang, Qing Gu
The paper investigates the learning dynamics of on‑policy distillation (OPD), a method that combines teacher supervision with training on student‑generated trajectories. It identifies two stages of the feedback loop: converting the distillation signal into parameter updates, and broadcasting those updates to alter predictions and reshape the rollout distribution. The study explains why students may be unresponsive to teacher disagreement and how updates that improve local agreement can lead to degeneration by increasing the likelihood of poor continuations. It also shows that offline distillation can prepare a student for OPD without boosting immediate accuracy, yet OPD can still yield significant accuracy gains after sufficient offline preparation.
By Guan Zhe Hong, Sumeet Ramesh Motwani, Liang Chen, Yangchen Pan, Junchi Yu, Jialin Yu, Philip Torr, Yi Ren
arXiv:2605. 03677v2 Announce Type: replace Abstract: On-policy distillation (OPD) has recently emerged as an effective post-training paradigm for consolidating the capabilities of specialized expert models into a single student model.
By Wenjin Hou, Shangpin Peng, Weinong Wang, Zheng Ruan, Yue Zhang, Zhenglin Zhou, Mingqi Gao, Yifei Chen, Kaiqi Wang, Hongming Yang, Chengquan Zhang, Zhuotao Tian, Han Hu, Yi Yang, Fei Wu, Hehe Fan
The paper introduces Stochastic Truncated On-Policy Distillation (STOPD), a method that speeds up on-policy distillation by randomly truncating student rollouts and correcting sampling bias with blockwise inverse-probability weighting. STOPD retains teacher feedback for every generated token and preserves the expected gradient of vanilla OPD while providing theoretical error bounds for common optimizers. Experiments show that STOPD cuts generated tokens by 25–48% and achieves 1.3–2.2× faster training without sacrificing accuracy.
By Zhengxin Zhang, Maosen Tang, Ning Wang, Changhao Li, Jingren Zhou, Bo Dai, Sainyam Galhotra, Alex Townsend, Claire Cardie
arXiv:2606. 00305v1 Announce Type: cross Abstract: On-Policy Distillation (OPD) improves large language model reasoning by training a student model on trajectories sampled from its own policy under teacher supervision.
By Yuxuan Jiang, Francis Ferraro
The paper proposes Mixture Proposal On-Policy Distillation (MPOPD) to address a mismatch issue in on‑policy distillation (OPD) where the rollout policy is fixed to the current student. By treating the rollout policy as an optimization target, MPOPD derives a geometric mixture of student and teacher policies and introduces a proposal policy for efficient rollout generation. Experiments on 16 reasoning benchmarks show consistent gains over OPD, notably a 6.91% accuracy improvement on math reasoning with Qwen3-4B-Base and a 41% reduction in generation length.
By Sirui Chen, Yunzhe Qi, Haobo Xu, Jiaru Zou, Mengting Ai, Zihao Li, Jingrui He
The paper introduces REOPOLD, a framework that improves on-policy distillation (OPD) for reasoning tasks by addressing the problem of uniform credit allocation. REOPOLD selectively controls where teacher feedback is applied, its strength, and the timing of changes during training, thereby reducing entropy collapse and negative transfer. Experiments show that REOPOLD achieves up to 12× higher sample efficiency compared to recent post‑training baselines and can be extended to cross‑vocabulary distillation, self‑distillation, and test‑time scaling.
By Jongwoo Ko, Sara Abdali, Young Jin Kim, Tianyi Chen, Pashmina Cameron
arXiv:2606. 21994v2 Announce Type: replace Abstract: On-policy distillation (OPD) improves reasoning models by applying dense teacher supervision on student-sampled trajectories.
By Qingfei Zhao, Huan Song, Shuyu Tian, Jiawei Shao, Xuelong Li
The paper introduces vOPD, an On‑Policy Distillation method that incorporates a control variate baseline from reinforcement learning to reduce gradient variance. By using the per‑token negative reverse KL divergence as a closed‑form value function, vOPD maintains a lightweight single‑sample estimator while keeping gradients unbiased. Experiments on mathematics, science, and instruction‑following tasks show vOPD consistently outperforms standard OPD and matches the performance of more expensive full‑vocabulary methods.
By Minjae Oh, Sangjun Song, Gyubin Choi, Yunho Choi, Yohan Jo