arXiv Machine Learning August 18, 2026 By Gordei Verbii, Juho Lee

The Distributional View of Knowledge Distillation

Read the original on arXiv Machine Learning →

arXiv:2608. 15215v1 Announce Type: cross Abstract: Token-level knowledge distillation (KD) matches two conditional distributions per position, yet the standard objectives compare them pointwise: a Kullback-Leibler gradient is blind to which wrong token receives probability mass.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

arXiv Computation and Language

4d ago

Distilling What Matters: Confidence-Aware Selective Distillation for Large Language Models

arXiv:2609.36734v1 Announce Type: new Abstract: Knowledge Distillation (KD) trains a smaller-capacity student model to imitate a larger-capacity teacher model by matching output distributions, implic...

By Ayan Sengupta, Vaibhav Seth, Tanmoy Chakraborty

llms efficiency benchmarks safety

arXiv AI

Jul 3

DemoPSD: Disagreement-Modulated Policy Self-Distillation

arXiv:2607. 02502v1 Announce Type: cross Abstract: On-policy self-distillation (OPSD) has emerged as a practical method for training large language models (LLMs) to reason, where a single model acts as both the teacher and the student with different levels of information access.

By Yunhe Li, Hao Shi, Wenhao Liu, Mengzhe Ruan, Hanxu Hou, Zhongxiang Dai, Shuang Qiu, Linqi Song

llms efficiency benchmarks

arXiv Machine Learning

Jun 11

Beyond Dark Knowledge: Mixup-Based Distillation for Reliable Predictions

arXiv:2606. 12171v1 Announce Type: cross Abstract: Knowledge Distillation (KD) and mixup have proven effective at inducing smoothness in class boundaries; KD captures inherent class relationships in probability distributions, and mixup enforces them through convex combinations of inputs.

By Jos\'e Medina, Paul Honeine, Abdelaziz Bensrhair, Amnir Hadachi

efficiency

arXiv Machine Learning

Aug 4

Distill What the Student Can See: Fisher-Projected On-Policy Distillation for Vision-Language Models

arXiv:2608. 01263v1 Announce Type: new Abstract: On-policy distillation (OPD) samples trajectories from the current student policy and minimizes token-level divergence between student and teacher next-token distributions at prefixes along those trajectories.

By Leyan Xue, Feng Xiong, Mingjun Ma, Changqing Zhang

llms efficiency multimodal benchmarks

arXiv Machine Learning

Sep 1

Influence-Directed Distillation: Solving the Diversity Bottleneck in Sampled-Token On-Policy Distillation

arXiv:2608.29846v1 Announce Type: cross Abstract: Sampled-token on-policy distillation (OPD) efficiently transfers capabilities from teacher to student using student-generated tokens, requiring teach...

By Run Yang, Runpeng Dai, Jie Sun, Jielei Zhang, Fan Zhou, Hongtu Zhu, Peiyi Li, Longwen Gao

efficiency

arXiv Machine Learning

Jul 9

Geometric Self-Distillation for Reasoning Generalization

arXiv:2607. 06855v1 Announce Type: new Abstract: On-policy distillation is a practical post-training recipe for large language models, supplying dense teacher supervision on the student's own trajectories.

By Josip Juki\'c, Ivan Titov

llms efficiency benchmarks

One email a morning, machine-written

Related stories

Distilling What Matters: Confidence-Aware Selective Distillation for Large Language Models

DemoPSD: Disagreement-Modulated Policy Self-Distillation

Beyond Dark Knowledge: Mixup-Based Distillation for Reliable Predictions

Distill What the Student Can See: Fisher-Projected On-Policy Distillation for Vision-Language Models

Influence-Directed Distillation: Solving the Diversity Bottleneck in Sampled-Token On-Policy Distillation

Geometric Self-Distillation for Reasoning Generalization