D‑Skill is a framework that distills the behavioral advantage of a large language‑model teacher into an interpretable natural‑language skill document, using only paired rollouts and no weight updates. The skill is constructed offline from fixed trajectories, then provided to a smaller student model at inference time. Across benchmarks such as SpreadsheetBench‑Verified, ALFWorld, and chess puzzles, D‑Skill closes 75‑95% of the teacher‑student performance gap and reduces inference cost by 5.7×–8.3× compared to the teacher.
By Shuo Yuan, Dacheng Li, Joseph E. Gonzalez
Selective Residual Adaptation (SRA) is a plug‑in framework that improves multi‑step reasoning in distilled large language models by correcting sparse distribution divergences between teacher and student along reasoning paths. It uses a Discrepancy Gate to spot divergence points and a Candidate Residual Adapter to calibrate top‑K logits, thereby avoiding changes to the student backbone or runtime teacher computation. Experiments on Qwen3 models from 0.6B to 4B show that SRA raises average Pass@16 by up to 4.37 percentage points while maintaining general capabilities.
By Zhenxing Dou, Kun He, Mingyang Yi, Yong Liu
KDFP is a new white‑box methodology for general knowledge distillation in large language models, addressing a gap left by prior work that focused mainly on post‑training abilities such as instruction following and chain‑of‑thought reasoning. The authors evaluate lessons from earlier studies and conduct new experiments to design a distillation approach tailored to modern LLMs. KDFP achieves 1.6%–4.9% higher performance across nine benchmarks and improves training efficiency by up to 99.1% through reduced temporary parameters.
By Ryan Swift, Konstantinos Psounis
The paper introduces Universal Textual Teaching (UTT), a parameter‑update‑free framework that distills knowledge gaps between a Teacher and Student large language model into a reusable natural‑language artifact called a Primer. UTT identifies representative gap cases, then iteratively refines the Primer through a cycle of student attempts, prompter‑generated teaching instructions, teacher demonstrations, and synthesizer consolidation. Experiments on math (Omni‑MATH‑2) and code generation (KernelBench) show significant accuracy gains, outperforming prompt engineering and parameter‑based knowledge distillation, and the Primer generalizes across different Teacher‑Student pairs.
By Zhanyi Lu, Huan Wang
The paper introduces masked self‑distillation, a post‑training framework that trains a language model to internalize portions of its own intermediate reasoning traces. By varying the fraction of trace internalized, the authors demonstrate that models can achieve higher inference efficiency and improved task performance on math and graph‑coloring problems. Experiments on Qwen3‑4B and Qwen3‑8B show that the method generalizes well to in‑domain out‑of‑distribution cases without catastrophic forgetting, and that supervised fine‑tuning alone can reduce trace length at the expense of generalization.
By Durgesh Kalwar, Vardhan Palod, Jaya Adithya Pavuluri, Subbarao Kambhampati
TraceForge introduces a two‑stage training method that generates reasoning traces in the style of a target model family, using supervised learning followed by adversarial refinement. The system produces traces that are judged as authentic with scores comparable to real traces, and when applied to teacher answers, it improves performance on coding benchmarks such as HumanEval+ and MBPP+. The approach addresses the limitation of existing models that only return answers without detailed reasoning, enabling more transparent and learnable outputs.
By Yan Wang
arXiv:2607. 06855v1 Announce Type: new Abstract: On-policy distillation is a practical post-training recipe for large language models, supplying dense teacher supervision on the student's own trajectories.
By Josip Juki\'c, Ivan Titov
arXiv:2607. 22629v1 Announce Type: new Abstract: Large Reasoning Models (LRMs) produce long, explicit chains of intermediate steps before generating a final answer at inference time.
By Durgesh Kalwar, Vardhan Palod, Subbarao Kambhampati
Bridged Self-Distillation (BSD) is a unified framework that enables on‑policy distillation from any teacher providing reasoning traces. The external teacher supplies a privileged reasoning trace offline, while a self‑teacher—an identical copy of the student conditioned on that trace—scores the student’s rollouts in its own token space. BSD uses these token‑level signals, along with a verifier, to adjust updates on unsuccessful rollouts, allowing the same training procedure to work with shared‑tokenizer, cross‑tokenizer, and closed‑API teachers and consistently outperforming scenario‑specific baselines.
By Runsheng Chen, Yihong Liu, Raoyuan Zhao, Michael A. Hedderich, Hinrich Sch\"utze, Matthias Grabmair
arXiv:2608.16647v2 Announce Type: replace
Abstract: On-policy distillation (OPD) transfers teacher capabilities by supervising trajectories sampled from the student's own policy, yet its generalizati...
By Zhaoyi Li, Deyang Kong, Yuan Wei, Evan Yang, Ranran Shen, Mahardika Krisna Ihsani, Ming Yang, Wei Zhang, Chuan Hao, Jian Yang, Ran Tao, Bryan Dai, Shikun Zhang, Wei Ye, Ying Wei, Defu Lian
The paper investigates whether a strong teacher is necessary for knowledge distillation in large language model pretraining. By varying teacher and student sizes and token budgets, the authors create strong‑to‑weak, same‑level, and weak‑to‑strong teacher‑student pairs and evaluate distillation effectiveness. They find that even small, undertrained teachers can improve larger students when losses are properly mixed, and that pushing teachers to be stronger can saturate or reverse gains, while distillation mainly boosts generalization rather than in‑domain fitting.
By Taiming Lu, Zhuang Liu
arXiv:2605.10641v2 Announce Type: replace-cross
Abstract: Large Vision-Language Models (VLMs) are successful in addressing a multitude of vision-language understanding tasks, such as Visual Question...
By Nikolaos Gkalelis, Vasileios Mezaris