Transferring Visual Explanations: How Cross-Architecture Knowledge Distillation Affects Model Interpretability
Read the original on arXiv Computer Vision →The Flow has not summarised this story yet — read it at arXiv Computer Vision.
The Flow has not summarised this story yet — read it at arXiv Computer Vision.
The paper investigates how knowledge distillation from Vision Transformers to smaller CNNs can cause dimensional collapse in the student’s representation space. Using SVD and Shannon entropy, the authors show that cosine‑based distillation leads to a drastic reduction in effective rank, while adding an InfoNCE objective can double the rank but harms downstream accuracy due to signal dilution. They further demonstrate that a label‑aware contrastive objective (Supervised Contrastive distillation) can maintain or improve accuracy without unnecessary rank expansion, indicating that effective rank alone is not a reliable indicator of representation quality.
Vision Transformers underperform convolutional networks when training data is scarce, and distilling convolutional inductive biases from a CNN teacher is an effective remedy that leaves the deployed model unchanged. General-purpose feature distillation, however, transfers little in this setting.
arXiv:2604.05819v2 Announce Type: replace-cross Abstract: Interpreting the decisions of complex computer vision models is crucial to establish trust and accountability, especially in safety-critical...
CalibCause is a quality‑gated, interpretable deep learning framework for diabetic retinopathy grading that integrates visual explanations, label‑free image‑quality conditioning, knowledge distillation, and calibrated confidence with deferral. Using 51,310 fundus images from multiple datasets, the distilled student model achieved a quadratic weighted kappa of 0.863 and accuracy of 0.865 on pooled testing, while maintaining comparable performance across held‑out environments. The framework also demonstrated improved calibration (expected calibration error 0.025–0.036) and higher accuracy (up to 0.923) when deferring the least confident 20% of images.
arXiv:2607. 22872v1 Announce Type: new Abstract: Post-training quantization (PTQ) has become a practical solution for deploying deep learning models on resource-constrained edge devices by compressing high-precision floating-point weights into low-precision representations without requiring retraining.
Post-training quantization (PTQ) has become a practical solution for deploying deep learning models on resource-constrained edge devices by compressing high-precision floating-point weights into low-precision representations without requiring retraining. Past research has demonstrated that quantization largely preserves classification accuracy; however, whether it also preserves the model's internal reasoning remains an open question.