DualOPSD introduces an asymmetric alternating framework for on-policy self-distillation, where a privileged teacher first trains the student and then adapts to the student’s updated distribution on the same trajectory, eliminating the need for additional rollouts. This adaptive approach improves average performance on AIME 2024, AIME 2025, and HMMT 2025 benchmarks for the Qwen3-8B model, with gains that vary by model scale. Across 1.7B, 4B, and 8B scales, DualOPSD also reduces truncation effects and lowers KL divergence between teacher and student.
By Yutong Chen, Guangfu Guo, Zhichao Xu, Kunpeng Liu
arXiv:2608. 09228v1 Announce Type: cross Abstract: On-Policy Self-Distillation (OPSD) is commonly interpreted as the transfer of privileged information: a teacher observes the verified solution to the target problem and supervises the student's trajectory.
By Yuki Ichihara, Naoto Iwase, Mohammad Atif Quamar, Junpei Komiyama
arXiv:2609.37915v1 Announce Type: new
Abstract: On-policy self-distillation (OPSD) trains a student to match a privileged teacher distribution along its own sampled trajectory. Standard OPSD applies...
By Md. Ismail Hossain, Humaira Kousar, Isidora Chara Tourni
arXiv:2608. 01735v2 Announce Type: replace Abstract: On-policy (self) distillation (OPSD) is increasingly adopted for language-model post-training.
By Jianyu Wu, Yizhou Wang, Encheng Su, Chen Tang, Shixiang Tang
arXiv:2609.36546v1 Announce Type: cross
Abstract: On-policy distillation (OPD) trains a student model on its self-generated trajectories with dense token-level teacher feedback. However, naive OPD ma...
By Shutong Wu, Xiwen Chen, Brendan Rappazzo, Daiheng Zhang, Anderson Schneider, Yuriy Nevmyvaka, Jiawei Zhang
arXiv:2608. 09447v1 Announce Type: cross Abstract: On-policy distillation (OPD) aligns a student with a teacher on trajectories sampled from the student itself, reducing the train-test state mismatch of offline distillation.
By Zehao Chen, Gongxun Li, Tianxiang Ai, Yifei Li, Zixuan Huang, Wang Zhou, Tao Huang, Fuzhen Zhuang, Xianglong Liu, Jianxin Li, Deqing Wang, Yikun Ban
arXiv:2609.08798v1 Announce Type: new
Abstract: Weak-to-strong generalization asks whether stronger models can learn from weaker supervisors and surpass them. This question is particularly important...
By Youngrok Park, Sangmin Bae, Hojung Jung, Jongwoo Ko, Yunseon Choi, Young Jin Kim, Pashmina Cameron, Aaron Courville, Se-Young Yun
arXiv:2607. 18293v1 Announce Type: new Abstract: On-policy self-distillation (OPSD) teaches large language models new skills through a teacher that shares the student's backbone and supervises its own rollouts.
By Yingzi Ma, Zichen Zhu, Ming Jiang, Chaowei Xiao
The paper investigates how different types of privileged context affect policy drift in on‑policy self‑distillation (OPSD) for language models. By varying content (demonstration, feedback, rephrase) and source (external, self‑generated with a verifier, self‑generated without a verifier) across nine combinations, the authors train Qwen2.5‑7B on three datasets and measure target‑task accuracy, prior‑task retention, reverse KL divergence, and parameter‑update geometry. Results show that changing content leads to a larger median KL range and more aligned parameter updates than changing source, indicating that privileged context significantly influences the direction and magnitude of policy updates.
By Ravenor Davion, Nick Rui
arXiv:2608. 05219v1 Announce Type: new Abstract: Privileged on-policy distillation provides dense supervision for multi-turn agents by allowing a synchronized teacher to re-score the student's response at every turn with access to training-only references, such as successful trajectories.
By Junzhuo Liu, Weiwei Li, Jun Ling, Peng Wang
arXiv:2608. 09826v1 Announce Type: cross Abstract: Reinforcement learning with verifiable rewards yields no group-relative signal when rollout groups are uniformly correct or uniformly wrong, which account for 63.
By Yubo Jiang, Fengying Xie, Zhiguo Jiang, Haopeng Zhang
arXiv:2610.14001v1 Announce Type: cross
Abstract: On-policy self-distillation (OPSD) uses a self-teacher conditioned on privileged information to turn the sparse outcome signal of reinforcement learn...
By Yuhang Yang, Mingxuan Xia, Chao Ye, Shuai Zhu, Shouqing Yang, Haobo Wang, Shenzhi Yang, Zhengqing Zang, Chenglong Wang