arXiv Computation and Language
SelFusion: Self-distillation for Diffusion Language Models
Read the original on arXiv Computation and Language →The Flow has not summarised this story yet — read it at arXiv Computation and Language.
Related stories
arXiv AI
Jun 2
ASKD-Whisper: Adaptive Self-knowledge Distillation for Efficient and Low-Latency Automatic Speech Recognition
arXiv:2601. 19919v2 Announce Type: replace-cross Abstract: Knowledge distillation (KD) is one of the most effective paradigms for compressing large-scale foundation models into deployable architectures.
arXiv Machine Learning
Jul 7
Few-Step Diffusion Language Models via Trajectory Self-Distillation
arXiv:2602. 12262v4 Announce Type: replace-cross Abstract: Diffusion large language models (DLLMs) have emerged as powerful generative models with the promise of fast text generation through parallel decoding.
arXiv Machine Learning
Aug 7
On-Policy Self-Distillation without Any Supervision
arXiv:2608. 06296v1 Announce Type: new Abstract: On-policy (Self-)Distillation (OPD / OPSD) has shown strong potential for post-training large language models (LLMs).
arXiv Machine Learning
Sep 15
Temporal Self-Distillation: Faster Inference in Discrete Diffusion Language Models
arXiv:2609.15177v1 Announce Type: new Abstract: Diffusion language models (dLLMs) promise fast inference by generating multiple tokens in parallel, but suffer severe performance degradation when para...
arXiv Machine Learning
5d ago
KDFP: A first-principles approach to knowledge distillation in large language models
KDFP is a new white‑box methodology for general knowledge distillation in large language models, addressing a gap left by prior work that focused mainly on post‑training abilities such as instruction following and chain‑of‑thought reasoning. The authors evaluate lessons from earlier studies and conduct new experiments to design a distillation approach tailored to modern LLMs. KDFP achieves 1.6%–4.9% higher performance across nine benchmarks and improves training efficiency by up to 99.1% through reduced temporary parameters.
Hugging Face Trending Papers
Aug 6
On-Policy Self-Distillation without Any Supervision
On-policy (Self-)Distillation (OPD / OPSD) has shown strong potential for post-training large language models (LLMs). However, existing methods still rely heavily on external supervision, including ground-truth signals, environmental feedback, or guidance from larger models, and therefore fall short of genuine "self"-distillation.