On-policy distillation (OPD) transfers teacher capabilities by supervising trajectories sampled from the student's own policy, yet its generalization behavior remains poorly understood, as most studies evaluate OPD on a single domain and on benchmarks close to the training data. We present a controlled study that varies one generalization factor at a time, from in-domain distribution shifts to cross-domain transfer and the multi-teacher setting.
arXiv:2608. 03632v1 Announce Type: new Abstract: On-Policy distillation (OPD) transfers teacher capabilities by supervising student-sampled trajectories with dense token-level teacher signals.
By Yinuo Jiang, Yongjie Ye, Zhou Tao, Xiang Zhuang, Qiang Zhang, Huajun Chen, Tiankai Li
On-Policy distillation (OPD) transfers teacher capabilities by supervising student-sampled trajectories with dense token-level teacher signals. Recent selective OPD methods improve this process by prioritizing signals that are confident, informative, or learnable.
arXiv:2609.37044v1 Announce Type: new
Abstract: Explicit intermediate reasoning gives large language models (LLMs) a stronger problem-solving mode. We study learning from this think-mode advantage vi...
By Wanqi Ren, Jianxiang Wang, Danxuan Liu, Linyi Ding, Yuan Zhang
arXiv:2607. 06855v1 Announce Type: new Abstract: On-policy distillation is a practical post-training recipe for large language models, supplying dense teacher supervision on the student's own trajectories.
By Josip Juki\'c, Ivan Titov
arXiv:2610.14022v1 Announce Type: cross
Abstract: On-policy distillation (OPD) improves language-model by supervising a student on prefixes sampled from its own policy, but its supervision remains co...
By Yingdong Shi, Zhihong Li, Changming Li, Jingyi Yu, Quan Li, Kan Ren
arXiv:2610.21234v1 Announce Type: new
Abstract: On-policy distillation (OPD) has recently been introduced as a promising paradigm for continual learning, where a student policy can be supervised at t...
By Stanley Wei, Jiin Woo, Kaan Ozkara, Youngsuk Park
arXiv:2607. 02502v1 Announce Type: cross Abstract: On-policy self-distillation (OPSD) has emerged as a practical method for training large language models (LLMs) to reason, where a single model acts as both the teacher and the student with different levels of information access.
By Yunhe Li, Hao Shi, Wenhao Liu, Mengzhe Ruan, Hanxu Hou, Zhongxiang Dai, Shuang Qiu, Linqi Song
The paper introduces Multi-Teacher Self-Distillation Policy Optimization (MT‑SDPO), an on‑policy distillation method that combines multiple frozen teachers into a single student model. MT‑SDPO uses self‑anchors, answer‑verified eligibility, and privileged distillation to select reliable teachers per sample rather than per domain. Experiments on five students from three model families show that MT‑SDPO improves the weakest domain of Qwen3‑8B by 14.79 points and reduces its domain gap by 74.7%, achieving a more balanced performance than matching a single teacher to each domain.
By Xixiang He, Xingming Li, Baiqi Wu, Qiyao Sun, Xuanyu Ji, Ao Cheng, Qingyong Hu
arXiv:2609.38025v1 Announce Type: cross
Abstract: On-policy distillation (OPD) trains a student on its own generated responses using dense, token-level supervision from a stronger teacher. Vanilla OP...
By Zhenyu Wang, Tianze Wang, Linjun Zhang, Yifan Hu
The paper investigates data efficiency and selection in On‑Policy Distillation (OPD) for large language models. It shows that 1‑shot OPD—training on a single example—consistently improves performance, especially when the example is hard, and that longer chain‑of‑thought (CoT) paths drive the gains rather than token entropy. Based on these findings, the authors propose a simple hard‑example selection strategy that, using only eight carefully chosen hard examples, matches the performance of a 17,000‑example baseline across models from 1.5B to 7B parameters.
By Zhinan Hou, Jiaqi Zhang, Xunliang Cai, Keyou You
The paper investigates on‑policy self‑distillation (OPSD) as a method to enhance reasoning in language models, focusing on mathematical reasoning across models from 0.6B to 8B parameters. Through controlled experiments and token‑level analysis, the authors find that OPSD’s effectiveness depends on alignment between the teacher’s reasoning mode and the full teacher prefix, rather than on privileged semantics alone. They observe that OPSD only improves reasoning in limited compatibility regimes, while often causing length growth, degradation, or behavioral collapse, and that the teacher’s signal is unstable and not predictive of downstream performance.
By Yang Li, Gongle Xue, Yuheng Yuan, Yijia Guo, Shizhe Zhang, Liwen Hu, Lei Ma