The paper introduces Tail‑Corrected Top‑k On‑Policy Distillation (TT‑OPD), a method that improves on existing Top‑k OPD by combining the selected top‑k tokens with a sampled token from the student’s rollout. This hybrid approach recovers the probability mass discarded by limiting to top‑k, yielding an unbiased estimator of the reverse KL divergence gradient while maintaining low computational cost. Experiments show TT‑OPD outperforms other OPD variants in accuracy.
By Linjian Meng, Siyuan Gan, YuHan Li, Xiran Wang, Ziyang Ding, Ditang Gou, Yiming Wu, Zhen Zhao
arXiv:2606. 17199v1 Announce Type: cross Abstract: Standard on-policy distillation (OPD) for large language models estimates the reverse-KL objective using student-sampled tokens, yielding an unbiased single-sample Monte Carlo estimator that avoids vocabulary-wide computation.
By Anhao Zhao, Junlong Tong, Yingqi Fan, Ping Nie, Wenjie Li, Xiaoyu Shen
arXiv:2605. 03677v2 Announce Type: replace Abstract: On-policy distillation (OPD) has recently emerged as an effective post-training paradigm for consolidating the capabilities of specialized expert models into a single student model.
By Wenjin Hou, Shangpin Peng, Weinong Wang, Zheng Ruan, Yue Zhang, Zhenglin Zhou, Mingqi Gao, Yifei Chen, Kaiqi Wang, Hongming Yang, Chengquan Zhang, Zhuotao Tian, Han Hu, Yi Yang, Fei Wu, Hehe Fan
arXiv:2604. 00626v4 Announce Type: replace Abstract: As Large Language Models continue to grow in both capability and cost, transferring frontier capabilities into smaller, deployable students has become an important engineering problem, and knowledge distillation remains a common technique for this transfer.
By Mingyang Song, Mao Zheng
arXiv:2609.37500v1 Announce Type: new
Abstract: On-policy distillation (OPD) trains language models using dense token-level teacher supervision on student-generated trajectories. However, its relianc...
By Yuxiao Yang, Shangzhe Li, Tianrun Yu, Kaixiang Zhao, Taylor W. Killian, Weitong Zhang
arXiv:2602. 22495v3 Announce Type: replace-cross Abstract: Reinforcement learning (RL) post-training has recently driven major gains in long chain-of-thought reasoning large language models (LLMs), but the high inference cost of such models motivates distillation into smaller students.
By Zhaoyang Zhang, Shuli Jiang, Yantao Shen, Yuting Zhang, Dhananjay Ram, Shuo Yang, Zhuowen Tu, Wei Xia, Stefano Soatto
RISE (Recursive Improvement via Self-Extrapolating Policy Distillation) is a new method that builds a synthetic teacher from a language model’s own RLVR training trajectory. By extrapolating the displacement between the current checkpoint and a trailing anchor in parameter or logit space, RISE transforms sparse outcome-based updates into dense token-level targets without external models or privileged conditioning. The approach recursively refines the student model, combining RLVR and on‑policy distillation, and demonstrates superior performance across mathematical reasoning, STEM, code generation, and multi‑turn agentic tasks.
By Yang Li, Semih Yavuz, Shafiq Joty
arXiv:2606. 24143v1 Announce Type: new Abstract: On-policy distillation (OPD) trains a student on its own rollouts guided by teacher feedback and is becoming increasingly important for large language model (LLM) post-training.
By Wonjun Kang, Kevin Galim, Seunghyuk Oh, Minjun Kang, Sanghyun Park, Donghoon Kim, Minjae Lee, Minseo Kim, Rishabh Tiwari, Yuchen Zeng, Hyung Il Koo, Kangwook Lee
arXiv:2607. 28582v1 Announce Type: new Abstract: On-policy self-distillation (OPSD) is a promising approach to improve reasoning language models, but it remains brittle in practice: making it work reliably often requires substantial engineering effort.
By Jiawei Xu, Minghui Liu, Juzheng Zhang, Tom Goldstein, Furong Huang
arXiv:2601. 07155v3 Announce Type: replace-cross Abstract: Knowledge distillation (KD) is a widely adopted technique for transferring knowledge from large language models to smaller student models; however, conventional supervised KD often suffers from a distribution mismatch between training and inference.
By Ijun Jang, Jewon Yeom, Juan Yeo, Hyunggyu Lim, Taesup Kim
arXiv:2604. 10688v2 Announce Type: replace-cross Abstract: On-policy reinforcement learning has become the dominant paradigm for reasoning alignment in large language models, yet its sparse, outcome-level rewards make token-level credit assignment notoriously difficult.
By Binbin Zheng, Xing Ma, Yiheng Liang, Jingqing Ruan, Xiaoliang Fu, Kepeng Lin, Benchang Zhu, Ke Zeng, Xunliang Cai
Reinforcement learning with verifiable rewards (RLVR) is a powerful recipe for improving language-model reasoning, but it is expensive to repeat on every new strong model because the target model must generate many rollouts during training. As models scale, post-training itself becomes a bottleneck.