arXiv:2606. 22600v2 Announce Type: replace-cross Abstract: On-Policy Distillation (OPD) improves the learning efficiency of standard reinforcement learning through dense, token-level supervision from teachers.
By Yan Xie, Sijie Zhu, Tiansheng Wen, Bo Chen, Yifei Wang
arXiv:2609.37500v1 Announce Type: new
Abstract: On-policy distillation (OPD) trains language models using dense token-level teacher supervision on student-generated trajectories. However, its relianc...
By Yuxiao Yang, Shangzhe Li, Tianrun Yu, Kaixiang Zhao, Taylor W. Killian, Weitong Zhang
The paper introduces Advantage-Based On-Policy Distillation (AOPD), a method that selectively weights teacher actions during online policy distillation by their counterfactual advantage to the student. Unlike standard OPD, which follows the teacher at every state, AOPD reinforces only those teacher actions that improve the student's expected outcome, using an estimator that relies solely on standard student rollouts and terminal rewards. Experiments on math, coding, and Medqa tasks demonstrate that AOPD outperforms both distillation and reinforcement learning baselines in external-teacher and self-distillation settings.
By Mahdi Sabbaghi, George Pappas, Hamed Hassani, Adel Javanmard
arXiv:2609.38025v1 Announce Type: cross
Abstract: On-policy distillation (OPD) trains a student on its own generated responses using dense, token-level supervision from a stronger teacher. Vanilla OP...
By Zhenyu Wang, Tianze Wang, Linjun Zhang, Yifan Hu
The paper introduces Tail‑Corrected Top‑k On‑Policy Distillation (TT‑OPD), a method that improves on existing Top‑k OPD by combining the selected top‑k tokens with a sampled token from the student’s rollout. This hybrid approach recovers the probability mass discarded by limiting to top‑k, yielding an unbiased estimator of the reverse KL divergence gradient while maintaining low computational cost. Experiments show TT‑OPD outperforms other OPD variants in accuracy.
By Linjian Meng, Siyuan Gan, YuHan Li, Xiran Wang, Ziyang Ding, Ditang Gou, Yiming Wu, Zhen Zhao
On-policy distillation (OPD) aligns a student model with a teacher's logit distribution on student-generated trajectories. This approach has achieved strong empirical gains and can often surpass conve...