arXiv:2607. 02234v1 Announce Type: new Abstract: On-policy self-distillation (OPSD) has emerged as a promising paradigm for improving LLM reasoning, where a privileged teacher with access to reference solutions provides token-level supervision on the student's own generated trajectories.
By Zhanming Shen, Jintao Tong, Shaotian Yan, Chen Shen, Hao Chen, Wentao Ye, Xiaomeng Hu, Rui Miao, Haobo Wang, Junbo Zhao, Gang Chen, Jieping Ye
The paper investigates on‑policy self‑distillation (OPSD), where a student model learns from its own outputs using token‑level supervision conditioned on privileged reference information. Experiments with Qwen3 models on science and mathematics datasets show that the correct reference does not consistently improve performance; students can improve without it, and solutions from other problems sometimes outperform the correct reference. The study finds that student predictions align more closely with the base model’s reasoning than with the reference supervision, and that alignment alone does not reliably predict performance gains.
By Samyak Shrestha, Alexander Tessier
arXiv:2607. 02502v1 Announce Type: cross Abstract: On-policy self-distillation (OPSD) has emerged as a practical method for training large language models (LLMs) to reason, where a single model acts as both the teacher and the student with different levels of information access.
By Yunhe Li, Hao Shi, Wenhao Liu, Mengzhe Ruan, Hanxu Hou, Zhongxiang Dai, Shuang Qiu, Linqi Song
arXiv:2608. 08176v1 Announce Type: new Abstract: On-policy self-distillation (OPSD) improves the reasoning abilities of LLMs by internalizing privileged context into model parameters through self-distillation.
By Yongkang Yang, Zhezheng Hao, Hong Zhang, Yi Liu, Xiankun Lin, Wence Ji, Fanjunduo Wei, Jiarui Yu, Qiang Lin, Xiaoyun Liang, Hande Dong
On-policy Distillation (OPD) supervises a student model on trajectories sampled from its own policy by minimizing the divergence between the output distributions of the teacher and student at each token position, thereby providing dense token-level supervision. Although existing OPD methods have demonstrated strong performance in improving the reasoning ability of student models, their objectives fundamentally rely on token-level distribution matching.
E$^2$-OPSD tackles a failure mode in on‑policy self‑distillation where the student’s token entropy surpasses the teacher’s, a problem called entropy overshoot. The method replaces the teacher’s answer with a retrieved solved neighboring problem to provide transferable reasoning guidance and uses the student‑teacher entropy gap to adjust token corrections. These changes improve math reasoning scores by up to 4.3 points in mean@16 and yield out‑of‑domain gains of up to 4.9 points in mean@16 and 5.5 points in pass@8, all without extra forward passes or networks.
By Yifei Liu, Minghao Fang, Xinyu Gu, Chengkai Yao, Mengdi Liu, Tengfei Ma, Jiangbin Zheng, Chang Yu, Zhangyang Gao