The paper investigates how knowledge distillation affects the internal mechanisms of language models by comparing GPT‑2 with its six‑layer student, DistilGPT‑2. Using techniques such as linear CKA, logit lens analysis, and attention pattern decomposition, the authors examine two circuits: Indirect Object Identification (IOI) and Greater‑Than (GT). They find that the IOI circuit collapses completely in the distilled model, whereas the GT circuit largely survives, with the student’s layer‑4 head effectively remapping the teacher’s layer‑9 comparison head, achieving 87.7% of the original task performance.
By Nabeelah Faumi, Deshan Sumanathilaka
KDFP is a new white‑box methodology for general knowledge distillation in large language models, addressing a gap left by prior work that focused mainly on post‑training abilities such as instruction following and chain‑of‑thought reasoning. The authors evaluate lessons from earlier studies and conduct new experiments to design a distillation approach tailored to modern LLMs. KDFP achieves 1.6%–4.9% higher performance across nine benchmarks and improves training efficiency by up to 99.1% through reduced temporary parameters.
By Ryan Swift, Konstantinos Psounis
arXiv:2605.06597v3 Announce Type: replace-cross
Abstract: Self-distillation (SD) offers a promising path for adapting large language models (LLMs) without relying on stronger external teachers. Howev...
By Yiqiao Jin, Yiyang Wang, Lucheng Fu, Yijia Xiao, Yinyi Luo, Haoxin Liu, B. Aditya Prakash, Chao Zhang, Josiah Hester, Jindong Wang, Srijan Kumar
arXiv:2610.14022v1 Announce Type: cross
Abstract: On-policy distillation (OPD) improves language-model by supervising a student on prefixes sampled from its own policy, but its supervision remains co...
By Yingdong Shi, Zhihong Li, Changming Li, Jingyi Yu, Quan Li, Kan Ren
Bridged Self-Distillation (BSD) is a unified framework that enables on‑policy distillation from any teacher providing reasoning traces. The external teacher supplies a privileged reasoning trace offline, while a self‑teacher—an identical copy of the student conditioned on that trace—scores the student’s rollouts in its own token space. BSD uses these token‑level signals, along with a verifier, to adjust updates on unsuccessful rollouts, allowing the same training procedure to work with shared‑tokenizer, cross‑tokenizer, and closed‑API teachers and consistently outperforming scenario‑specific baselines.
By Runsheng Chen, Yihong Liu, Raoyuan Zhao, Michael A. Hedderich, Hinrich Sch\"utze, Matthias Grabmair
arXiv:2607. 09692v1 Announce Type: new Abstract: Model distillation -- training on outputs from stronger third-party models -- is widely used to boost performance, but raises concerns about unfair advantages and policy violations.
By Rajat Rawat, Sizhe Chen, Akshay Anand, Michael Duan, Bob Rotsted, Sewon Min
The paper introduces Localized Context Distillation (LoCD), a method that improves learning from interaction experience for large language models. LoCD identifies critical spans in a student’s trajectory that need revision, then has a teacher regenerate only those spans from the student’s own prefix, reducing exposure bias while focusing on informative states. Experiments across math reasoning, code generation, and agent tasks show that LoCD outperforms strong baselines and yields more stable training with smaller, sparser updates.
By Zhezheng Hao, Yongkang Yang, Xilin Xia, Jiarui Yu, Yi Liu, Fanjunduo Wei, Peijie Dong, Tianfu Wang, Qiang Lin, Can Wang, Hande Dong, Jiawei Chen
Activation-Conditioned Self-Distillation (ACSD) is a new on‑policy self‑distillation method that uses a frozen copy of the base model to extract a steering vector by contrasting activations from self‑generated trajectories that reach verified correct answers with all other trajectories. The student learns from next‑token distributions on its own prefixes, without needing reference text or teacher parameter updates, and is used alone at inference. Across five models, ACSD achieves the highest mean accuracy on four mathematical benchmarks, with notable gains on DeepSeek‑R1‑0528‑Qwen3‑8B and LiveCodeBench v6 compared to the OPSD baseline.
By Zhexi Lu, Subhajit Chaudhury, Tejaswini Pedapati, Keerthiram Murugesan, Lei Yu
arXiv:2605.10194v2 Announce Type: replace
Abstract: On-policy self-distillation (OPSD) uses a model as its own teacher under privileged context, providing token-level supervision on the model's own r...
By Jiaxuan Wang, Xuan Ouyang, Zhiyu Chen, Yulan Hu, Lan-Zhe Guo
arXiv:2606. 09456v1 Announce Type: new Abstract: On-Policy Distillation (OPD) has become a core technique in the post-training of Large Language Models (LLMs) for transferring knowledge from domain experts to student models.
By Yifan Niu, Han Xiao, Dongyi Liu, Zelong Wang, Dihong Gong, Yasheng Wang, Jia Li
On-policy distillation can transfer the reasoning capabilities of a strong teacher to a smaller student. However, distillation from a stronger teacher model faces two mismatches: its solutions may be...
arXiv:2605.10889v2 Announce Type: replace-cross
Abstract: On-policy distillation offers dense, per-token supervision for training reasoning models; however, it remains unclear under which conditions...
By Mohammadreza Armandpour, Fatih Ilhan, David Harrison, Ajay Jaiswal, Duc N. M Hoang, Fartash Faghri, Yizhe Zhang, Minsik Cho, Mehrdad Farajtabar