The paper introduces Common‑Support Distillation (CSD), a logits‑based framework for transferring knowledge across different tokenizers by constructing a shared support between teacher and student distributions. CSD handles both one‑to‑one alignments using explicit common‑token pairs and other alignments by creating a virtual support via the autoregressive chain rule, while a residual event ensures normalization. Experiments on Llama‑3.2‑1B, Gemma3‑1B‑PT, Qwen, and Phi models show that CSD consistently outperforms existing cross‑tokenizer methods, achieving up to 1.8‑point gains over the best alternative and surpassing both continued pretraining and same‑tokenizer distillation.
By Mingyu Yang, Sharath Turuvekere Sreenivas, Adithyakrishna Venkatesh Hanasoge, Nima Tajbakhsh, Ashwath Aithal, Pavlo Molchanov
arXiv:2607. 22334v1 Announce Type: new Abstract: Open-weight language models from different families exhibit complementary capabilities, motivating their consolidation into a compact student through on-policy distillation (OPD).
By Hao Wang, Kun Yuan, Wenlin Zhong, Minglei Zhang, Han Xiao, Ming Sun, Honggang Qi
arXiv:2610.08448v1 Announce Type: cross
Abstract: On-Policy Distillation (OPD) trains a student on its own generations using teacher feedback. With different tokenizers, comparing teacher and student...
By Bingxi Hou, Guochao Jiang, Guofeng Quan, Weiqing Li, Wenfeng Feng, Guohua Liu, Yuewei Zhang
SimCT is a method for on‑policy distillation that addresses the problem of mismatched tokenizers between teacher and student models. By comparing teacher and student predictions over short multi‑token continuations that both tokenizers can produce, SimCT restores supervision that would otherwise be lost when only exact shared tokens are used. Experiments on mathematical reasoning and code‑generation tasks show consistent improvements over shared‑vocabulary OPD and other cross‑tokenizer baselines, with ablations confirming the gains come from recovering discarded supervision.
By Jie Sun, Mao Zheng, Mingyang Song, Qiyong Zhong, Yilin Cheng, Bichuan Feng, Pengfei Liu, Junfeng Fang, Xiang Wang
arXiv:2608. 09836v1 Announce Type: new Abstract: On-policy distillation (OPD) has emerged as a core component of modern LLM post-training pipelines, yet we reveal a failure mode: degenerate agreement, where students exploit repetitive loops to achieve near-perfect token agreement with the teacher despite globally flawed responses.
By Zichao Yu, Chengzhi Yu, Shengze Xu, Yujin Han, Bingqing Jiang, Xu Wang, Difan Zou
arXiv:2609.17474v1 Announce Type: cross
Abstract: Large language model (LLM) distillation aims to transfer the capabilities of a powerful teacher to a smaller student. Direct imitation, however, can...
By Haichen Hu, Yuheng Zhang, David Simchi-Levi
The paper introduces Adaptive Local Relational Alignment (ALRA), a logit‑based knowledge distillation method for autoregressive language models that combines student‑generated token proposals with teacher guidance at each prediction position. ALRA dynamically selects the number of candidate tokens based on the teacher’s probability spread, uses Adaptive Local Divergence to match both mass and relative token distributions, and applies Student‑Weighted Pairwise Relational Alignment to focus on high‑probability token pairs. Experiments on The Pile show that 200M‑ and 500M‑parameter students trained with ALRA outperform the best baseline by roughly 1 percentage point and surpass pre‑training without distillation by over 2 percentage points on nine zero‑shot benchmarks.
By Quang Hoang Trung, Quang Huu Hieu, Nguyen Van Hoang Phuc, Vo Nguyen Le Duy
The paper introduces REOPOLD, a framework that improves on-policy distillation (OPD) for reasoning tasks by addressing the problem of uniform credit allocation. REOPOLD selectively controls where teacher feedback is applied, its strength, and the timing of changes during training, thereby reducing entropy collapse and negative transfer. Experiments show that REOPOLD achieves up to 12× higher sample efficiency compared to recent post‑training baselines and can be extended to cross‑vocabulary distillation, self‑distillation, and test‑time scaling.
By Jongwoo Ko, Sara Abdali, Young Jin Kim, Tianyi Chen, Pashmina Cameron
arXiv:2505.12781v5 Announce Type: replace-cross
Abstract: Training high-performing Small Language Models (SLMs) remains costly, even with knowledge distillation and pruning from larger teacher models...
By Jitai Hao, Qiang Huang, Hao Liu, Xinyan Xiao, Zhaochun Ren, Jun Yu
The paper introduces Iterative Calibration and Learning (IterCaL), a method for large language model distillation that mitigates teacher bias when target-domain reward feedback is unavailable. IterCaL alternates between calibrating the source-reward teacher via token-level branching and training the student on target prompts, achieving polynomial convergence to an oracle student. The authors prove that regularized direct teacher matching cannot reach the oracle, whereas IterCaL converges to it using only source rewards.
By Haichen Hu, Yuheng Zhang, David Simchi-Levi
The paper investigates how the sampled-token reverse-KL loss in on‑policy distillation distributes updates across tokens. By analyzing the gradient of the per‑token K2 estimator, the authors find that tokens with low student probability and large teacher‑student gaps receive disproportionately large gradient norms. They propose Surprise‑aware Reweighting (SuRe), a lightweight weighting rule that further amplifies this allocation, and demonstrate that SuRe improves math metrics on Qwen3 student models without harming out‑of‑domain performance.
By Bing Shao, Jiazheng Zhang, Long Ma, Yujiong Shen, Senjie Jin, Xin Guo, Yuming Yang, Mingxu Chai, Zhiheng Xi, Tao Gui, Qi Zhang, Xuanjing Huang
arXiv:2608. 14277v1 Announce Type: cross Abstract: On-policy distillation (OPD) offers a promising way to transfer reasoning capabilities from stronger teacher models, but applying it to long-context reasoning teachers and short-context students introduces practical challenges, including tokenizer mismatch, teacher-student distribution mismatch, response length explosion, and training instability.
By Haonan He, Haodi Lei, Yun Luo, Haoran Zhang, Shunkai Zhang, Yizhuo Li, Shengji Tang, Zhilin Wang, Runzhe Zhan, Lei Bai, Ganqu Cui, Fangchen Yu, Yafu Li, Peng Ye, Ning Ding, Yu Cheng