arXiv Machine Learning June 11, 2026 By Jos\'e Medina, Paul Honeine, Abdelaziz Bensrhair, Amnir Hadachi

Beyond Dark Knowledge: Mixup-Based Distillation for Reliable Predictions

Read the original on arXiv Machine Learning →

arXiv:2606. 12171v1 Announce Type: cross Abstract: Knowledge Distillation (KD) and mixup have proven effective at inducing smoothness in class boundaries; KD captures inherent class relationships in probability distributions, and mixup enforces them through convex combinations of inputs.

Summary generated by The Flow from the publisher's feed. The full article lives at arXiv Machine Learning.

efficiency

arXiv AI

Aug 11

Distill Skills into Weights, Not Prompts: Abstract Skills as Privileged Signals for On-Policy Self-Distillation

arXiv:2608. 09826v1 Announce Type: cross Abstract: Reinforcement learning with verifiable rewards yields no group-relative signal when rollout groups are uniformly correct or uniformly wrong, which account for 63.

By Yubo Jiang, Fengying Xie, Zhiguo Jiang, Haopeng Zhang

reinforcement-learning efficiency benchmarks

Hugging Face Trending Papers

Aug 10

Distill Skills into Weights, Not Prompts: Abstract Skills as Privileged Signals for On-Policy Self-Distillation

Reinforcement learning with verifiable rewards yields no group-relative signal when rollout groups are uniformly correct or uniformly wrong, which account for 63. 0-68.

reinforcement-learning efficiency benchmarks

arXiv Machine Learning

Aug 4

Distill What the Student Can See: Fisher-Projected On-Policy Distillation for Vision-Language Models

arXiv:2608. 01263v1 Announce Type: new Abstract: On-policy distillation (OPD) samples trajectories from the current student policy and minimizes token-level divergence between student and teacher next-token distributions at prefixes along those trajectories.

By Leyan Xue, Feng Xiong, Mingjun Ma, Changqing Zhang

llms efficiency multimodal benchmarks

arXiv AI

Jul 7

Covert Trait Propagation Is Representation Alignment: Mechanistic Evidence from Hidden-Channel Distillation

arXiv:2607. 04432v1 Announce Type: cross Abstract: A student model trained on pure uniform noise can still inherit its teacher's digit-classification ability, provided the two share initialization.

By Kargi Chauhan, Aditya Shah

llms efficiency safety

arXiv Machine Learning

1d ago

The Distributional View of Knowledge Distillation

arXiv:2608. 15215v1 Announce Type: cross Abstract: Token-level knowledge distillation (KD) matches two conditional distributions per position, yet the standard objectives compare them pointwise: a Kullback-Leibler gradient is blind to which wrong token receives probability mass.

By Gordei Verbii, Juho Lee

rag fine-tuning efficiency

arXiv Machine Learning

Jul 14

Student Capacity Moderates Knowledge Distillation Effectiveness: A Systematic Study Across ResNet Teacher-Student Pairs on CIFAR-10

arXiv:2605. 31191v2 Announce Type: replace Abstract: We investigate how teacher-student capacity relationships modulate knowledge distillation (KD) effectiveness in ResNet-based image classification on CIFAR-10.

By Umut Onur Yasar

computer-vision efficiency

Related stories

Distill Skills into Weights, Not Prompts: Abstract Skills as Privileged Signals for On-Policy Self-Distillation

Distill Skills into Weights, Not Prompts: Abstract Skills as Privileged Signals for On-Policy Self-Distillation

Distill What the Student Can See: Fisher-Projected On-Policy Distillation for Vision-Language Models

Covert Trait Propagation Is Representation Alignment: Mechanistic Evidence from Hidden-Channel Distillation

The Distributional View of Knowledge Distillation

Student Capacity Moderates Knowledge Distillation Effectiveness: A Systematic Study Across ResNet Teacher-Student Pairs on CIFAR-10