Knowledge distillation (KD) improves language-model pretraining by matching teacher distributions at corpus prefixes, but provides no direct supervision at contexts induced by student predictions. We...
The paper introduces On-Policy Mid-Training Distillation (OPMD), a method that replaces the standard next-token prediction (NTP) objective in continual pre‑training with an active, query‑driven approach. By sampling trajectories from the evolving student policy and applying dense token‑level reverse‑KL rewards from a frozen teacher, OPMD trains the model on states it actually visits, reducing the mismatch between training and generation. Experiments on a 1B‑token mathematical corpus show that OPMD improves performance for the Qwen3‑1.7B‑Base model on MATH‑500 and Minerva benchmarks, outperforming the NTP baseline that degrades on both tasks, while adding no inference‑time overhead.
The paper introduces On-Policy Mid-Training Distillation (OPMD), a method that shifts continual pre‑training from passive next‑token prediction to active, query‑driven knowledge acquisition. By sampling trajectories from the evolving student policy and applying dense token‑level reverse‑KL rewards from a frozen teacher, OPMD trains the model on states it actually visits, reducing the mismatch between training and generation. Experiments on MATH‑500 and Minerva show that OPMD improves a Qwen student’s performance over its base and outperforms the standard next‑token prediction baseline, while integrating seamlessly into existing continual pre‑training pipelines without inference‑time overhead.
By Ruize Zheng, Xuyang Liu, Siyu Ren, Xiaobing Tu, Yinggui Wang, Jinkui Ren, Xiantao Zhang, Linfeng Zhang
arXiv:2606. 08432v1 Announce Type: new Abstract: On-policy distillation (OPD) has become a central post-training tool for large language models (LLMs), providing dense per-token teacher supervision along the student's own rollouts.
By Li Jiang, Haoran Xu, Yichuan Ding, Amy Zhang
arXiv:2609.37500v1 Announce Type: new
Abstract: On-policy distillation (OPD) trains language models using dense token-level teacher supervision on student-generated trajectories. However, its relianc...
By Yuxiao Yang, Shangzhe Li, Tianrun Yu, Kaixiang Zhao, Taylor W. Killian, Weitong Zhang
arXiv:2606. 09456v1 Announce Type: new Abstract: On-Policy Distillation (OPD) has become a core technique in the post-training of Large Language Models (LLMs) for transferring knowledge from domain experts to student models.
By Yifan Niu, Han Xiao, Dongyi Liu, Zelong Wang, Dihong Gong, Yasheng Wang, Jia Li
arXiv:2610.08448v1 Announce Type: cross
Abstract: On-Policy Distillation (OPD) trains a student on its own generations using teacher feedback. With different tokenizers, comparing teacher and student...
By Bingxi Hou, Guochao Jiang, Guofeng Quan, Weiqing Li, Wenfeng Feng, Guohua Liu, Yuewei Zhang
arXiv:2608. 07935v1 Announce Type: new Abstract: On-policy self-distillation (OPSD) adapts a language model by distilling guidance from a frozen teacher on trajectories sampled from the student.
By Meilin Yang (Renmin University of China, Beijing, China), Zixuan Ding (Renmin University of China, Beijing, China), Jianhao Nie (Renmin University of China, Beijing, China), Weite Zhang (Renmin University of China, Beijing, China), Yuxin Zhang (Renmin University of China, Beijing, China), Zhiming Shao (Renmin University of China, Beijing, China), Li Yu (Renmin University of China, Beijing, China), Zhe Fu (Renmin University of China, Beijing, China)
Activation-Conditioned Self-Distillation (ACSD) is a new on‑policy self‑distillation method that uses a frozen copy of the base model to extract a steering vector by contrasting activations from self‑generated trajectories that reach verified correct answers with all other trajectories. The student learns from next‑token distributions on its own prefixes, without needing reference text or teacher parameter updates, and is used alone at inference. Across five models, ACSD achieves the highest mean accuracy on four mathematical benchmarks, with notable gains on DeepSeek‑R1‑0528‑Qwen3‑8B and LiveCodeBench v6 compared to the OPSD baseline.
By Zhexi Lu, Subhajit Chaudhury, Tejaswini Pedapati, Keerthiram Murugesan, Lei Yu
arXiv:2609.33455v2 Announce Type: replace
Abstract: On-policy distillation (OPD) trains a student model on its own trajectories using dense token-level feedback from a stronger teacher model. Since e...
By Zizhuo Lin, Quanling Liu, Yi Yang, Yawei Luo
arXiv:2605. 03677v2 Announce Type: replace Abstract: On-policy distillation (OPD) has recently emerged as an effective post-training paradigm for consolidating the capabilities of specialized expert models into a single student model.
By Wenjin Hou, Shangpin Peng, Weinong Wang, Zheng Ruan, Yue Zhang, Zhenglin Zhou, Mingqi Gao, Yifei Chen, Kaiqi Wang, Hongming Yang, Chengquan Zhang, Zhuotao Tian, Han Hu, Yi Yang, Fei Wu, Hehe Fan
arXiv:2606. 21994v2 Announce Type: replace Abstract: On-policy distillation (OPD) improves reasoning models by applying dense teacher supervision on student-sampled trajectories.
By Qingfei Zhao, Huan Song, Shuyu Tian, Jiawei Shao, Xuelong Li