arXiv Computer Vision

Transferring Visual Explanations: How Cross-Architecture Knowledge Distillation Affects Model Interpretability

arXiv Computer Vision
Sep 3

Breaking the Geometric Bottleneck: Contrastive Expansion in Asymmetric Cross-Modal Distillation

The paper investigates how knowledge distillation from Vision Transformers to smaller CNNs can cause dimensional collapse in the student’s representation space. Using SVD and Shannon entropy, the authors show that cosine‑based distillation leads to a drastic reduction in effective rank, while adding an InfoNCE objective can double the rank but harms downstream accuracy due to signal dilution. They further demonstrate that a label‑aware contrastive objective (Supervised Contrastive distillation) can maintain or improve accuracy without unnecessary rank expansion, indicating that effective rank alone is not a reliable indicator of representation quality.

By Kabir Thayani
Hugging Face Trending Papers
Aug 11

Grid-Preserving Knowledge Distillation: Transferring Convolutional Inductive Bias to Vision Transformers under Data Scarcity

Vision Transformers underperform convolutional networks when training data is scarce, and distilling convolutional inductive biases from a CNN teacher is an effective remedy that leaves the deployed model unchanged. General-purpose feature distillation, however, transfers little in this setting.

arXiv Machine Learning
Nov 2

CalibCause: A Quality-Gated, Interpretable and Calibrated Deep Learning Framework for Diabetic Retinopathy Grading

CalibCause is a quality‑gated, interpretable deep learning framework for diabetic retinopathy grading that integrates visual explanations, label‑free image‑quality conditioning, knowledge distillation, and calibrated confidence with deferral. Using 51,310 fundus images from multiple datasets, the distilled student model achieved a quadratic weighted kappa of 0.863 and accuracy of 0.865 on pooled testing, while maintaining comparable performance across held‑out environments. The framework also demonstrated improved calibration (expected calibration error 0.025–0.036) and higher accuracy (up to 0.923) when deferring the least confident 20% of images.

By Sohaib Sulman, Muhammad Atif Saeed, Akhtar Jamil, Alaa Ali Hameed, Roland Aydin
arXiv Machine Learning
Jul 28

Same Predictions, Different Reasons: The Effect of Quantization on Model Explanations

arXiv:2607. 22872v1 Announce Type: new Abstract: Post-training quantization (PTQ) has become a practical solution for deploying deep learning models on resource-constrained edge devices by compressing high-precision floating-point weights into low-precision representations without requiring retraining.

By Kazi Kamruzzaman Rabbi, Md. Zami Al Zunaed Farabe, M. Sohel Rahman
Hugging Face Trending Papers
Jul 24

Same Predictions, Different Reasons: The Effect of Quantization on Model Explanations

Post-training quantization (PTQ) has become a practical solution for deploying deep learning models on resource-constrained edge devices by compressing high-precision floating-point weights into low-precision representations without requiring retraining. Past research has demonstrated that quantization largely preserves classification accuracy; however, whether it also preserves the model's internal reasoning remains an open question.

arXiv Computer Vision
Oct 27

Promise and Limits of Cross-Architecture Knowledge Distillation for Lightweight Medical Vision Transformers (LiteMed-ViT)

arXiv:2610.33520v1 Announce Type: new Abstract: Running deep models for medical image analysis at the point of care is challenging with standard architectures: convolutional neural networks (CNNs) ar...

By Shuaibu Shehu (SR University), Elijah M. Flomo (SR University), Mohammad Safiya (GITAM University), Anurodh Kumar (SR University), Subodh Kumar (Dr. KNS Memorial Institute of Medical Sciences)
arXiv AI
Aug 26

Too much of a good thing -- when knowledge distillation promotes overfitting, and how to avoid it

The paper investigates how knowledge distillation (KD) applied at intermediate layers of a neural network can affect overfitting and model performance. While traditional KD focuses on the final output, this study explores block‑wise KD across eleven datasets, finding that on standard datasets the last block suffices, but on fine‑grained, data‑scarce settings intermediate supervision significantly improves accuracy. The authors also analyze optimal supervision granularity using attention maps, Centered Kernel Alignment, and Grad‑CAM, and examine teacher‑student fine‑tuning strategies.

By Irene Trigueros-Lorca, Leonardo Concepci\'on, Christian Wagner, Isaac Triguero, Daniel Molina
arXiv AI
Aug 11

UniDFKD: A Unified Semantic Prior Framework for Architecture-Agnostic Data-Free Knowledge Distillation

arXiv:2608. 09287v1 Announce Type: cross Abstract: Data-Free Knowledge Distillation (DFKD) transfers knowledge from a pretrained teacher model to a compact student model by synthesizing semantically informative data, eliminating the need for access to the original training dataset.

By Xuewan He, Tong Chu, Zihan Cheng, Yuchen Su, Qianxin Xia, Guoming Lu, Jielei Wang, Wen Li