arXiv AI August 26, 2026 By Irene Trigueros-Lorca, Leonardo Concepci\'on, Christian Wagner, Isaac Triguero, Daniel Molina

Too much of a good thing -- when knowledge distillation promotes overfitting, and how to avoid it

Read the original on arXiv AI →

The paper investigates how knowledge distillation (KD) applied at intermediate layers of a neural network can affect overfitting and model performance. While traditional KD focuses on the final output, this study explores block‑wise KD across eleven datasets, finding that on standard datasets the last block suffices, but on fine‑grained, data‑scarce settings intermediate supervision significantly improves accuracy. The authors also analyze optimal supervision granularity using attention maps, Centered Kernel Alignment, and Grad‑CAM, and examine teacher‑student fine‑tuning strategies.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

arXiv AI

Jul 9

Enhancing deep learning models for time series classification via knowledge distillation

arXiv:2607. 06796v1 Announce Type: cross Abstract: Deep learning has achieved remarkable success in various domains including time series analysis, computer vision and natural language processing.

By Javidan Abdullayev, Maxime Devanne, Jonathan Weber, Germain Forestier

llms computer-vision nlp efficiency benchmarks

arXiv Machine Learning

Sep 15

Discovering and Preserving Category Correlation Knowledge via Adaptive Reciprocal Knowledge Distillation

arXiv:2609.13199v1 Announce Type: new Abstract: Knowledge distillation aims to improve the performance of lightweight student models by transferring knowledge from larger and more powerful teacher mo...

By Dawen Jiang, Zhishu Shen, Zeyu Liu, Tiehua Zhang

efficiency benchmarks safety

Hugging Face Trending Papers

Aug 11

Grid-Preserving Knowledge Distillation: Transferring Convolutional Inductive Bias to Vision Transformers under Data Scarcity

Vision Transformers underperform convolutional networks when training data is scarce, and distilling convolutional inductive biases from a CNN teacher is an effective remedy that leaves the deployed model unchanged. General-purpose feature distillation, however, transfers little in this setting.

llms efficiency benchmarks safety

Hugging Face Trending Papers

Jun 22

Understanding Knowledge Distillation in Post-Training: When It Helps and When It Fails

Large language models (LLMs) achieve strong performance across many tasks, but their high computational cost limits deployment in resource-constrained environments. Knowledge Distillation (KD) offers a practical solution by transferring knowledge from a teacher model of a larger size to a smaller student model.

llms fine-tuning efficiency

arXiv Machine Learning

Oct 2

Model-to-Data Distillation for Graph Neural Networks

The paper introduces Model-to-Data (M2D) distillation, a method that transfers properties learned by a complex graph neural network (GNN) teacher into the graph data itself. By jointly learning augmented node features and graph structure, M2D encodes the teacher’s behavior, allowing simpler GNNs to recover high predictive performance, fairness, and robustness. The resulting graph can be reused with various downstream models, enabling them to approximate sophisticated teachers such as fairness-aware GNNs, Graph Attention Networks, and Graph Transformers.

By Debolina Halder Lina, Arlei Silva

llms efficiency safety

arXiv AI

Jun 30

FD$^2$: A Dedicated Framework for Fine-Grained Dataset Distillation

arXiv:2603. 25144v2 Announce Type: replace-cross Abstract: Dataset distillation (DD) compresses a large training set into a small synthetic set, reducing storage and training cost, and has shown strong results on general benchmarks.

By Hongxu Ma, Guang Li, Shijie Wang, Dongzhan Zhou, Baoli Sun, Takahiro Ogawa, Miki Haseyama, Zhihui Wang

efficiency benchmarks