arXiv Machine Learning July 14, 2026 By Umut Onur Yasar

Student Capacity Moderates Knowledge Distillation Effectiveness: A Systematic Study Across ResNet Teacher-Student Pairs on CIFAR-10

Read the original on arXiv Machine Learning →

arXiv:2605. 31191v2 Announce Type: replace Abstract: We investigate how teacher-student capacity relationships modulate knowledge distillation (KD) effectiveness in ResNet-based image classification on CIFAR-10.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

arXiv Machine Learning

Aug 11

A Controlled Study of Feature-Based Knowledge Distillation Across Student Designs

arXiv:2608. 08294v1 Announce Type: new Abstract: Knowledge distillation trains a smaller student to match the outputs of a larger teacher.

By Abhinand Balachandran, Praveen Prashant

efficiency

arXiv Machine Learning

Sep 22

Do Student LLMs Inherit OOD Robustness? Invariance-Weighted Distillation for Reliable Knowledge Transfer

arXiv:2609.22566v1 Announce Type: cross Abstract: Knowledge distillation (KD) aims to compress high-performance teacher LLMs into lightweight students. However, distilled students often exhibit subst...

By Dileesha Kannangara, Sanghamitra Dutta

llms nlp efficiency benchmarks

arXiv Machine Learning

Jul 30

When Top-K Misses the Decision: Tool-Call Drift in Multi-Teacher On-Policy Distillation

arXiv:2607. 07050v3 Announce Type: replace-cross Abstract: Top-K teacher logits make on-policy distillation tractable, but probability mass is not the same as decision support.

By Jiabin Shen, Guang Chen, Chengjun Mao

efficiency safety

arXiv Machine Learning

Sep 15

Discovering and Preserving Category Correlation Knowledge via Adaptive Reciprocal Knowledge Distillation

arXiv:2609.13199v1 Announce Type: new Abstract: Knowledge distillation aims to improve the performance of lightweight student models by transferring knowledge from larger and more powerful teacher mo...

By Dawen Jiang, Zhishu Shen, Zeyu Liu, Tiehua Zhang

efficiency benchmarks safety

arXiv Machine Learning

Jun 25

Distill on a Diet: Efficient Knowledge Distillation via Learnable Data Pruning

arXiv:2606. 25488v1 Announce Type: new Abstract: Knowledge Distillation (KD) is widely used to obtain compact models for efficient inference in resource-constrained environments.

By Yifan Wu, Yiqi Wang, Xichen Ye, Wenjing Yan, Xiaoqiang Li, Cheng Jin, Xiangyu Yue, Weizhong Zhang

efficiency

arXiv Machine Learning

2d ago

Learning Beyond Full Imitation: Task-Preserving Knowledge Distillation

arXiv:2609.39338v1 Announce Type: new Abstract: Knowledge distillation transfers knowledge by encouraging a student to match a teacher's predicted class probabilities. These probabilities express not...

By Qianfeng Yuan, Wenbing Tao

efficiency

One email a morning, machine-written

Related stories

A Controlled Study of Feature-Based Knowledge Distillation Across Student Designs

Do Student LLMs Inherit OOD Robustness? Invariance-Weighted Distillation for Reliable Knowledge Transfer

When Top-K Misses the Decision: Tool-Call Drift in Multi-Teacher On-Policy Distillation

Discovering and Preserving Category Correlation Knowledge via Adaptive Reciprocal Knowledge Distillation

Distill on a Diet: Efficient Knowledge Distillation via Learnable Data Pruning

Learning Beyond Full Imitation: Task-Preserving Knowledge Distillation