The paper introduces Multi-Teacher Self-Distillation Policy Optimization (MT‑SDPO), an on‑policy distillation method that combines multiple frozen teachers into a single student model. MT‑SDPO uses self‑anchors, answer‑verified eligibility, and privileged distillation to identify the most reliable teacher for each sample rather than relying on domain labels. Experiments on five students from three model families show that MT‑SDPO improves the weakest domain of Qwen3‑8B by 14.79 points and reduces its domain gap by 74.7%, achieving a better balance than matching a single teacher per domain.
The paper introduces Multi-Teacher Self-Distillation Policy Optimization (MT‑SDPO), an on‑policy distillation method that combines multiple frozen teachers into a single student model. MT‑SDPO uses self‑anchors, answer‑verified eligibility, and privileged distillation to select reliable teachers per sample rather than per domain. Experiments on five students from three model families show that MT‑SDPO improves the weakest domain of Qwen3‑8B by 14.79 points and reduces its domain gap by 74.7%, achieving a more balanced performance than matching a single teacher to each domain.
By Xixiang He, Xingming Li, Baiqi Wu, Qiyao Sun, Xuanyu Ji, Ao Cheng, Qingyong Hu
The paper investigates three fusion paradigms—Merge, Mix RL, and multi‑teacher on‑policy distillation (MOPD)—for consolidating reinforcement learning with verifiable rewards (RLVR) across multiple domains. Experiments across model scales and a multi‑domain benchmark show that while overall performance differences are small, significant gaps can appear on specific tasks, and each method exhibits distinct training dynamics and constraints. Practical guidelines are offered: Merge for cheap fusion when experts exist, Mix RL for unified training with adjustable domain mixtures, and MOPD when preserving domain‑specific gains is paramount.
By Siye Wu, Kai Yang, Yuchen Cai, Xin Xu, Peng-Yuan Wang, Jiaxuan Wang, Jiashun Liu, Jiafei Lyu, Yangkun Chen, Saiyong Yang, Yanghua Xiao
PMOPD introduces a projection-based approach to multi-teacher on-policy distillation, addressing the capability seesaw problem by constructing subspace memories from task-specific parameter displacements and projecting gradients and optimizer updates to avoid cross-task interference. It also includes a lightweight conflict probe for task interaction analysis, a task ordering strategy, and a cycling mechanism to balance subspace estimation and task revisitation. Experiments on Code, Reason, and Math tasks demonstrate that PMOPD improves all evaluated capabilities, raising average scores by 2.54 points on Qwen2.5-7B and 2.09 points on Llama-3.1-8B.
arXiv:2607. 26246v1 Announce Type: new Abstract: On-policy distillation (OPD), which aligns a student with the teacher's token-level distribution on the student's own rollouts, is an effective paradigm for transferring capabilities across LLMs.
By Fangxu Yu, Zinan Lin, Xiaodong Liu, Weijia Xu, Michael Xu, Tianyi Zhou, Jianfeng Gao
arXiv:2609.08798v1 Announce Type: new
Abstract: Weak-to-strong generalization asks whether stronger models can learn from weaker supervisors and surpass them. This question is particularly important...
By Youngrok Park, Sangmin Bae, Hojung Jung, Jongwoo Ko, Yunseon Choi, Young Jin Kim, Pashmina Cameron, Aaron Courville, Se-Young Yun