The paper investigates how privileged information—such as a teacher’s full solution or reasoning trace—affects on‑policy self‑distillation (OPSD) in language models. Using the AMPLE‑Math benchmark, the authors compare distillation with and without extra teacher views, finding that reference‑free distillation explains most gains for Qwen3‑1.7B, while additional references provide modest benefits, especially for polished solutions. The study also shows that the impact of privileged data depends on the student’s training regime and that altering token‑level supervision can leave student behavior largely unchanged.
By XiuYu Zhang, Wei Chow, Junfeng Fang, Zhenkai Liang, Tat-Seng Chua
The paper introduces Monte Carlo Privileged Distillation (MCPD) to address reference-induced target instability in on-policy self-distillation (OPSD). MCPD models privileged context as a distribution over partial reference views and averages teacher distributions across these views, yielding a target that matches the expected KL divergence minimization. Experiments on mathematical and scientific reasoning benchmarks show that MCPD provides more reference-consistent supervision and consistently outperforms full-reference OPSD.
By Khouloud Saadi, Di Wang
arXiv:2607. 02234v1 Announce Type: new Abstract: On-policy self-distillation (OPSD) has emerged as a promising paradigm for improving LLM reasoning, where a privileged teacher with access to reference solutions provides token-level supervision on the student's own generated trajectories.
By Zhanming Shen, Jintao Tong, Shaotian Yan, Chen Shen, Hao Chen, Wentao Ye, Xiaomeng Hu, Rui Miao, Haobo Wang, Junbo Zhao, Gang Chen, Jieping Ye
The paper introduces a recursive self-improvement framework for language models that replaces an external teacher with a frozen copy of the student, enabling dynamic co-evolution (DCE) and self-refined concise learning (SRCL). DCE allows the privileged teacher to evolve alongside the student, while SRCL trains on shorter, verified rewrites to reduce verbosity. Experiments show that the combined DCE+SRCL approach outperforms traditional on‑policy self‑distillation across multiple model sizes and math benchmarks, achieving significant accuracy gains and shorter outputs.
By Shangjian Yin, Zehao Zhao, Kavosh Asadi, Rui Liu, Yuchen Lu, Shike Mei, Hang Cui, Luke Simon, Zhouxing Shi, Hamed Firooz
The paper introduces On-Policy Attention Self-Distillation (OPASD), a method that augments token-level supervision with solution-conditioned attention distillation for reasoning models. OPASD projects a privileged teacher’s attention onto student-visible positions, renormalizes the distribution, and aligns it with the student. Experiments on three model sizes and four math benchmarks show that OPASD improves accuracy by 4.98–8.40 percentage points, reduces generated tokens by 73.9%, cuts compute by 72.6%, and trains 1.53× faster compared to token-only distillation.
By Safaeid Hossain Arib, Rabeya Akter, Ismam Nur Swapnil, Md. Faiyaz Abdullah Sayeedi, Tasnim Mohiuddin, Md Mofijul Islam
TeachBack is a bidirectional preference-learning framework for on‑policy self‑distillation that teaches a language model not only from a privileged Teacher but also how the Teacher should teach. By controlling the length of the reference‑reasoning suffix and varying prompt style, TeachBack encourages the Teacher to explore and re‑solve problems, reducing reliance on the reference solution while maintaining comparable accuracy. Experiments on mathematical and logical reasoning tasks demonstrate that TeachBack consistently improves Student performance over existing OPSD methods and related baselines.
By Linfei Ji (Huazhong University of Science and Technology), Jiayun Li (Tsinghua University), Haozhao Wang (Huazhong University of Science and Technology), Siyu Jiang (Huazhong University of Science and Technology), Zehao Wang (Harbin Institute of Technology), Longtao Xiao (Huazhong University of Science and Technology), Yu-Jie Yuan (Huawei Technologies Ltd), Ruixuan Li (Huazhong University of Science and Technology)