arXiv Machine Learning
Temperature-Adaptive Transformed Teacher Matching
Read the original on arXiv Machine Learning →The Flow has not summarised this story yet — read it at arXiv Machine Learning.
Related stories
arXiv Machine Learning
Sep 15
Discovering and Preserving Category Correlation Knowledge via Adaptive Reciprocal Knowledge Distillation
arXiv:2609.13199v1 Announce Type: new Abstract: Knowledge distillation aims to improve the performance of lightweight student models by transferring knowledge from larger and more powerful teacher mo...
arXiv Computer Vision
Aug 26
IDeaL: Data-Free Multi-Teacher Distillation via Improved Dead Leaves
The paper introduces IDeaL, a data‑free multi‑teacher distillation technique that generates teacher‑specific, improved samples using decorrelation losses at patch and image levels. By tailoring noise to each teacher, IDeaL produces strong student models that capture complementary teacher information and achieve results close to those distilled from real images. Experiments demonstrate that with only 1,000 images, students trained on IDeaL samples match or exceed the performance of students distilled from a 1,000‑image subset of ImageNet.
arXiv Machine Learning
Aug 18
Rethinking Reverse KL as Adaptive Entropy Distillation
arXiv:2608. 14685v1 Announce Type: new Abstract: Knowledge distillation (KD) is widely used to transfer the capabilities of large language models (LLMs) to smaller students, but existing objectives often struggle to balance faithful imitation and robust generation.
arXiv AI
Jun 24
ARIA: Adaptive Region-Based Importance Allocation for Conditional Diffusion Distillation
arXiv:2606. 23898v1 Announce Type: cross Abstract: Distilling conditional diffusion models aims to transfer the behavior of a large teacher to a smaller student while preserving alignment across conditioning inputs.
arXiv Machine Learning
Sep 1
Influence-Directed Distillation: Solving the Diversity Bottleneck in Sampled-Token On-Policy Distillation
arXiv:2608.29846v1 Announce Type: cross Abstract: Sampled-token on-policy distillation (OPD) efficiently transfers capabilities from teacher to student using student-generated tokens, requiring teach...
arXiv Machine Learning
Jun 11
Beyond Dark Knowledge: Mixup-Based Distillation for Reliable Predictions
arXiv:2606. 12171v1 Announce Type: cross Abstract: Knowledge Distillation (KD) and mixup have proven effective at inducing smoothness in class boundaries; KD captures inherent class relationships in probability distributions, and mixup enforces them through convex combinations of inputs.