arXiv:2609.38025v1 Announce Type: cross
Abstract: On-policy distillation (OPD) trains a student on its own generated responses using dense, token-level supervision from a stronger teacher. Vanilla OP...
By Zhenyu Wang, Tianze Wang, Linjun Zhang, Yifan Hu
arXiv:2606. 22600v2 Announce Type: replace-cross Abstract: On-Policy Distillation (OPD) improves the learning efficiency of standard reinforcement learning through dense, token-level supervision from teachers.
By Yan Xie, Sijie Zhu, Tiansheng Wen, Bo Chen, Yifei Wang
arXiv:2607. 04037v1 Announce Type: cross Abstract: On-policy distillation is a powerful way to transfer reasoning ability from a strong teacher to a smaller student: the student samples trajectories from its own policy, and the teacher provides dense token-level supervision on the states the student actually visits.
By Mohammad Sadegh Akhondzadeh, Vijay Lingam, Atula Tejaswi, Chanakya Ekbote, Sujay Sanghavi, Aleksandar Bojchevski
arXiv:2607. 28582v1 Announce Type: new Abstract: On-policy self-distillation (OPSD) is a promising approach to improve reasoning language models, but it remains brittle in practice: making it work reliably often requires substantial engineering effort.
By Jiawei Xu, Minghui Liu, Juzheng Zhang, Tom Goldstein, Furong Huang
The paper introduces Student-Delta On-Policy Distillation (SD-OPD), a method that refines language-model reasoning by using the student’s own pretrained base as a reference for teacher feedback. Unlike traditional on‑policy distillation (OPD), SD‑OPD scales updates based on centered student‑base differences, applies them only when they align with teacher‑student feedback, and recomputes the student signal each training round. Experiments with Qwen3 students ranging from 1.7B to 8B parameters show that SD‑OPD boosts six‑task mathematics average pass@1 by 7.76 to 10.37 percentage points over OPD.
By Bozhen Fang, Linhao Zhong, Xuehao Wang, Huanyi Zheng, Hao Chen, Chunhua Shen
arXiv:2608. 08176v1 Announce Type: new Abstract: On-policy self-distillation (OPSD) improves the reasoning abilities of LLMs by internalizing privileged context into model parameters through self-distillation.
By Yongkang Yang, Zhezheng Hao, Hong Zhang, Yi Liu, Xiankun Lin, Wence Ji, Fanjunduo Wei, Jiarui Yu, Qiang Lin, Xiaoyun Liang, Hande Dong
arXiv:2607. 26246v1 Announce Type: new Abstract: On-policy distillation (OPD), which aligns a student with the teacher's token-level distribution on the student's own rollouts, is an effective paradigm for transferring capabilities across LLMs.
By Fangxu Yu, Zinan Lin, Xiaodong Liu, Weijia Xu, Michael Xu, Tianyi Zhou, Jianfeng Gao
MetaOPD introduces a bilevel optimization framework that jointly learns a student model and a lightweight token‑weighting network for on‑policy distillation. The inner objective updates the student using weighted OPD, while the outer objective optimizes the weighting network based on validation loss after a virtual student update, enabling the token‑weight mapping to adapt to the student’s evolving needs. Experiments on multiple reasoning and out‑of‑domain datasets show that MetaOPD improves performance over standard OPD by up to 6.41 percentage points on larger student models.
By Zipeng Wang, Xinpeng Dong, Yuefan Wang, Pingchen Lu, Xian Wei, Kun Kuang, Fei Wu, Zhongxiang Dai, Min Zhang
The paper proposes Mixture Proposal On-Policy Distillation (MPOPD) to address a mismatch issue in on‑policy distillation (OPD) where the rollout policy is fixed to the current student. By treating the rollout policy as an optimization target, MPOPD derives a geometric mixture of student and teacher policies and introduces a proposal policy for efficient rollout generation. Experiments on 16 reasoning benchmarks show consistent gains over OPD, notably a 6.91% accuracy improvement on math reasoning with Qwen3-4B-Base and a 41% reduction in generation length.
By Sirui Chen, Yunzhe Qi, Haobo Xu, Jiaru Zou, Mengting Ai, Zihao Li, Jingrui He
The paper investigates on‑policy distillation (OPD), showing that teacher supervision during OPD contains significant noise that grows with teacher size, yet the student policy remains largely unaffected by this noise. It finds that OPD’s gains stem mainly from suppressing low‑log‑probability tokens, a process that can be replicated without a teacher. Building on this insight, the authors propose On‑Policy Self‑Adaptation (OPSA), a supervision‑free method that uses entropy‑adaptive negative advantages to improve performance on several benchmarks, outperforming both the base model and OPD.
By Yi Ding, Ruqi Zhang
arXiv:2607. 18955v1 Announce Type: new Abstract: Reinforcement learning with verifiable rewards (RLVR) has substantially improved the reasoning capabilities of large language models on tasks such as mathematical reasoning and code generation.
By Qiye Cai, Yichuan Ma, Linyang Li, Peiji Li, Yongkang Chen, Qipeng Guo, Yicheng Zou, Tao Gui, Xiaocheng Feng, Bing Qin
arXiv:2608. 19408v1 Announce Type: new Abstract: On-policy distillation (OPD) has emerged as an effective framework for post-training language models by pairing student-generated trajectories with dense token-level supervision from a teacher.
By Chen Yang, Haiyuan Wan, Rengrong Xiong, Yize Chen, Danny H. K. Tsang