The paper investigates how privileged information—such as a teacher’s full solution or reasoning trace—affects on‑policy self‑distillation (OPSD) in language models. Using the AMPLE‑Math benchmark, the authors compare distillation with and without extra teacher views, finding that reference‑free distillation explains most gains for Qwen3‑1.7B, while additional references provide modest benefits, especially for polished solutions. The study also shows that the impact of privileged data depends on the student’s training regime and that altering token‑level supervision can leave student behavior largely unchanged.
By XiuYu Zhang, Wei Chow, Junfeng Fang, Zhenkai Liang, Tat-Seng Chua
arXiv:2607. 05184v1 Announce Type: new Abstract: Self-distillation is a promising recipe for self-improvement in language models.
By Simran Kaur, Narutatsu Ri, Yinghui He, Liam Fowl, Sanjeev Arora
arXiv:2607. 22629v1 Announce Type: new Abstract: Large Reasoning Models (LRMs) produce long, explicit chains of intermediate steps before generating a final answer at inference time.
By Durgesh Kalwar, Vardhan Palod, Subbarao Kambhampati
arXiv:2605. 12400v2 Announce Type: replace-cross Abstract: We study on-policy self-distillation (OPSD), where a language model improves its reasoning ability by distilling privileged teacher distributions along its own on-policy trajectories.
By Yuxiao Yang, Xiaoyun Wang, Weitong Zhang
arXiv:2607. 02234v1 Announce Type: new Abstract: On-policy self-distillation (OPSD) has emerged as a promising paradigm for improving LLM reasoning, where a privileged teacher with access to reference solutions provides token-level supervision on the student's own generated trajectories.
By Zhanming Shen, Jintao Tong, Shaotian Yan, Chen Shen, Hao Chen, Wentao Ye, Xiaomeng Hu, Rui Miao, Haobo Wang, Junbo Zhao, Gang Chen, Jieping Ye
arXiv:2610.14022v1 Announce Type: cross
Abstract: On-policy distillation (OPD) improves language-model by supervising a student on prefixes sampled from its own policy, but its supervision remains co...
By Yingdong Shi, Zhihong Li, Changming Li, Jingyi Yu, Quan Li, Kan Ren
The paper investigates on‑policy self‑distillation (OPSD) as a method to enhance reasoning in language models, focusing on mathematical reasoning across models from 0.6B to 8B parameters. Through controlled experiments and token‑level analysis, the authors find that OPSD’s effectiveness depends on alignment between the teacher’s reasoning mode and the full teacher prefix, rather than on privileged semantics alone. They observe that OPSD only improves reasoning in limited compatibility regimes, while often causing length growth, degradation, or behavioral collapse, and that the teacher’s signal is unstable and not predictive of downstream performance.
By Yang Li, Gongle Xue, Yuheng Yuan, Yijia Guo, Shizhe Zhang, Liwen Hu, Lei Ma
arXiv:2606. 10385v1 Announce Type: cross Abstract: On-policy distillation (OPD) has demonstrated strong empirical gains in enhancing complex reasoning in LLMs by aligning a student model with a teacher's predictive distribution over the student's own trajectories.
By Wenhao Zhang
arXiv:2608. 19408v1 Announce Type: new Abstract: On-policy distillation (OPD) has emerged as an effective framework for post-training language models by pairing student-generated trajectories with dense token-level supervision from a teacher.
By Chen Yang, Haiyuan Wan, Rengrong Xiong, Yize Chen, Danny H. K. Tsang
The paper introduces Cognition-Aligned Reasoning Distillation (CARD), an RL-based method that converts solver trajectories from large reasoning models into teaching trajectories better suited for smaller student models. CARD uses two reward signals—a Distribution Alignment Reward and a Logical Coherence Reward—to align the teacher’s outputs with the student’s cognitive capabilities and reduce misleading reasoning steps. Experiments show that CARD improves both in-domain and out-of-distribution performance compared to distillation from raw teacher trajectories.
By Yashuo Luo, Qianren Mao, Ziqi Qin, Junnan Liu, Haitao Li, Qili Zhang, Yuan Gao, Zhijun Chen, Xu Wang
Waypoint-OPD is a new on-policy distillation method that leverages an intermediate teacher checkpoint to compare early and late policy shifts in reinforcement learning. By gating the endpoint shift based on the directional agreement between these early and late shifts, the method suppresses conflicting updates and requires only one additional frozen-teacher scoring pass. Experiments with Qwen3-4B teachers and larger Qwen3 student models in mathematical and code reasoning tasks show that Waypoint-OPD consistently outperforms endpoint-only transfer across all six domain–student settings.
By Bingxi Hou, Guochao Jiang, Guofeng Quan, Weiqing Li, Wenfeng Feng, Guohua Liu, Yuewei Zhang
Bootstrapped On-Policy Distillation (BOPD) is a two‑stage framework that first uses on‑policy self‑distillation (OPSD) to capture task‑relevant knowledge from reference‑conditioned supervision into an intermediate model, and then distills this intermediate model into a fresh copy of the initial model. The approach addresses the degradation observed in OPSD during continued training, which is caused by a growing mismatch between the evolving student and the frozen teacher. Experiments demonstrate that BOPD improves mathematical reasoning and code generation beyond both the initial models and the constructed OPSD teachers.
By Yue Wu, Xueyu Zhou, Yangrong Hu, Jian Huang