arXiv:2607. 17247v1 Announce Type: cross Abstract: Large language model (LLM) post-training is essential for improving reasoning, adaptation, and alignment.
By Chen Wang, Zhaochun Li, Jionghao Bai, Yining Zhang, Hexuan Deng, Ge Lan, Yue Wang
arXiv:2602. 22495v3 Announce Type: replace-cross Abstract: Reinforcement learning (RL) post-training has recently driven major gains in long chain-of-thought reasoning large language models (LLMs), but the high inference cost of such models motivates distillation into smaller students.
By Zhaoyang Zhang, Shuli Jiang, Yantao Shen, Yuting Zhang, Dhananjay Ram, Shuo Yang, Zhuowen Tu, Wei Xia, Stefano Soatto
arXiv:2605.10194v2 Announce Type: replace
Abstract: On-policy self-distillation (OPSD) uses a model as its own teacher under privileged context, providing token-level supervision on the model's own r...
By Jiaxuan Wang, Xuan Ouyang, Zhiyu Chen, Yulan Hu, Lan-Zhe Guo
The study compares supervised fine-tuning (SFT) and reinforcement learning (RL) as teacher training methods for on-policy distillation (OPD) in a single-teacher setting. Using Qwen3.5-9B models across Agentic, Reasoning, and Perception domains, RL-trained teachers consistently produce stronger students and better recovery of performance gains than SFT-trained teachers. The authors attribute this advantage to RL teachers’ smaller parameter shifts from the shared initialization, which facilitates learning through OPD.
By Jingyuan Huang, Zuming Huang, Yucheng Shi, Zhongzhi Li, Xiaoming Zhai, Wei Chu, Ninghao Liu
OPDSearch+ introduces a two‑stage distillation framework for search‑augmented reasoning that eliminates the need for task‑specific teacher fine‑tuning. In the first stage, a frozen off‑the‑shelf instruct model guides a student through live search interactions using a per‑position forward KL objective, transferring reasoning decomposition and evidence integration skills. The second stage refines this student with reinforcement learning, achieving performance surpassing RL alone and outperforming all prior 3B‑parameter baselines on seven QA benchmarks, including 13.1% improvement on HotpotQA and 8.5% on 2WikiMultihopQA.
By Qinglin Ye, Zhiyuan Gu, Jingjie Xia, Yiheng Zhang, Kaiyan Zhao, Shunchao Zheng, Yuhang Mu, Wenchao Du, Yiming Wang
Waypoint-OPD is a new on-policy distillation method that leverages an intermediate teacher checkpoint to compare early and late policy shifts in reinforcement learning. By gating the endpoint shift based on the directional agreement between these early and late shifts, the method suppresses conflicting updates and requires only one additional frozen-teacher scoring pass. Experiments with Qwen3-4B teachers and larger Qwen3 student models in mathematical and code reasoning tasks show that Waypoint-OPD consistently outperforms endpoint-only transfer across all six domain–student settings.
By Bingxi Hou, Guochao Jiang, Guofeng Quan, Weiqing Li, Wenfeng Feng, Guohua Liu, Yuewei Zhang