The paper investigates whether on‑policy distillation (OPD) is always advantageous for transferring knowledge from a teacher to a student model. It introduces Semi‑OPD, which uses offline rollouts from the initial student, and demonstrates that across 17 teacher‑student pairs (1.5B–235B parameters) Semi‑OPD outperforms OPD in 14 cases, achieving up to +13.6% accuracy and an 11.4× speedup. The authors show that the benefit of OPD depends on the alignment between teacher and student, measured by an output‑token overlap ratio, and argue that effective distillation requires on‑policy behavior relative to both models.
By Siyan Zhao, Yonggan Fu, Jindong Jiang, Shih-Yang Liu, Song Bian, Byung-Kwan Lee, Sharath Turuvekere Sreenivas, Wenliang Dai, Hanrong Ye, Aditya Grover, Pavlo Molchanov
The paper introduces SCOUT, a co‑training framework that adapts an off‑policy teacher to better continue from student‑generated prefixes in on‑policy distillation (OPD). By periodically optimizing the teacher’s conditional continuation ability using reinforcement learning with verifiable rewards, SCOUT improves the teacher’s performance on student prefixes. Experiments across various teacher‑student setups, model scales, and reasoning domains show that SCOUT consistently enhances the effectiveness of OPD.
By Langlin Huang, Hao Liu, Mononito Goswami, Xinyu Li, Prithwith Jana, Nikos Kanakaris, Patrick Bl\"obaum, Purak Jain
arXiv:2606. 09304v1 Announce Type: cross Abstract: On-policy distillation (OPD) trains a student on its own trajectories with dense per-token supervision from a stronger teacher, and often outperforms off-policy distillation and standard reinforcement learning.
By Haoran Xu, Hongyu Wang, Yifei Gao, Jiaze Li, Xiaofeng Zhang, Xiaosong Yuan
The paper investigates on‑policy distillation (OPD), a method that trains a student model using its own rollouts while a teacher remains frozen, to see if it can help a student improve on problems it currently fails. Experiments with Qwen‑3 and Phi‑4 models on a competition‑mathematics benchmark show that OPD actually degrades performance when training focuses on failure cases, even though training loss (KL divergence) continues to drop. However, when OPD is applied to a broader set of problems, it improves accuracy by 4.6 percentage points, indicating that OPD works well when most training data contains correct student prefixes but can reinforce errors if the training regime concentrates on failures.
By Shuo Yang, Zheyu Zhang, Mahito Sugiyama, Gjergji Kasneci
arXiv:2606. 22600v2 Announce Type: replace-cross Abstract: On-Policy Distillation (OPD) improves the learning efficiency of standard reinforcement learning through dense, token-level supervision from teachers.
By Yan Xie, Sijie Zhu, Tiansheng Wen, Bo Chen, Yifei Wang
The paper investigates on‑policy distillation (OPD), showing that teacher supervision during OPD contains significant noise that grows with teacher size, yet the student policy remains largely unaffected by this noise. It finds that OPD’s gains stem mainly from suppressing low‑log‑probability tokens, a process that can be replicated without a teacher. Building on this insight, the authors propose On‑Policy Self‑Adaptation (OPSA), a supervision‑free method that uses entropy‑adaptive negative advantages to improve performance on several benchmarks, outperforming both the base model and OPD.
By Yi Ding, Ruqi Zhang
On-policy distillation (OPD) aligns a student model with a teacher's logit distribution on student-generated trajectories. This approach has achieved strong empirical gains and can often surpass conve...
The paper introduces Teacher-Gated On-Policy Distillation (TGOPD), a method that verifies teacher reliability at the prompt level before applying dense supervision in on-policy distillation. TGOPD uses verifier-scored teacher probes to decide whether to route a prompt to dense OPD or to a verifier-grounded alternative. Experiments on 4B and 35B models across mathematics, code, and instruction tasks show TGOPD outperforms vanilla OPD and improves teacher GPU utilization from 9.8% to 78.9% in a 4B single-domain run.
By Zhiwei Zhang, Zechen Sun, Fei Zhao, Kang Peng, Bin Liang, Huayu Deng, Yao Hu, Kam-Fai Wong, Mu Chuan
arXiv:2608. 16333v1 Announce Type: cross Abstract: On-policy distillation (OPD) aligns a student model with a teacher's logit distribution on student-generated trajectories.
By Changhui Sun, Lanbo Liu, Hang Lei, Tong Ling, Jiahang Xie, Zhiyong Zheng, Yujia Wang, Hao Liu, Feng Xiao, Lu Liu, Yanlong Du, Zifeng Cheng, Ziwei Jiang, Qing Gu
arXiv:2609.08798v1 Announce Type: new
Abstract: Weak-to-strong generalization asks whether stronger models can learn from weaker supervisors and surpass them. This question is particularly important...
By Youngrok Park, Sangmin Bae, Hojung Jung, Jongwoo Ko, Yunseon Choi, Young Jin Kim, Pashmina Cameron, Aaron Courville, Se-Young Yun
CA-OPD is a confidence‑aware on‑policy distillation framework that improves structured visual prediction by using teacher confidence to selectively correct unreliable student transitions and gradually transfer rollout control to the student. The method aligns supervision with intervention decisions, providing direct cross‑entropy loss for corrected tokens and full predictive distribution for retained tokens. In a multi‑teacher setting for GUI grounding and OCR, CA‑OPD significantly outperforms the Qwen3.5‑0.8B baseline, achieving large gains on benchmarks such as ScreenSpot‑Pro and OCRBench‑v2 English.
By Menghao Li, Linjie Mu, Yin Wang, Haotian Hu, Yannian Gu, Lujiayi Xue, Fanyi Wang
The paper introduces TISD, a trajectory-intervention self-distillation method that forces a teacher-selected branch action and then lets the student generate the suffix, distilling the full trajectory under a privileged-context-conditioned teacher. This approach addresses a data-collection bottleneck in on‑policy self‑distillation by exposing successor contexts that the student would otherwise miss. Experiments on coding and science domains show modest but consistent improvements in average performance metrics compared to baseline methods.
By Taeckyung Lee, Rinat Amankos, Jeonghye Kim, Hyungjun Yoon, Woogyeol Jin, Sung-Ju Lee