arXiv Computer Vision
Sep 3

Breaking the Geometric Bottleneck: Contrastive Expansion in Asymmetric Cross-Modal Distillation

The paper investigates how knowledge distillation from Vision Transformers to smaller CNNs can cause dimensional collapse in the student’s representation space. Using SVD and Shannon entropy, the authors show that cosine‑based distillation leads to a drastic reduction in effective rank, while adding an InfoNCE objective can double the rank but harms downstream accuracy due to signal dilution. They further demonstrate that a label‑aware contrastive objective (Supervised Contrastive distillation) can maintain or improve accuracy without unnecessary rank expansion, indicating that effective rank alone is not a reliable indicator of representation quality.

By Kabir Thayani
Hugging Face Trending Papers
Aug 11

Grid-Preserving Knowledge Distillation: Transferring Convolutional Inductive Bias to Vision Transformers under Data Scarcity

Vision Transformers underperform convolutional networks when training data is scarce, and distilling convolutional inductive biases from a CNN teacher is an effective remedy that leaves the deployed model unchanged. General-purpose feature distillation, however, transfers little in this setting.

arXiv Machine Learning
Nov 2

CalibCause: A Quality-Gated, Interpretable and Calibrated Deep Learning Framework for Diabetic Retinopathy Grading

CalibCause is a quality‑gated, interpretable deep learning framework for diabetic retinopathy grading that integrates visual explanations, label‑free image‑quality conditioning, knowledge distillation, and calibrated confidence with deferral. Using 51,310 fundus images from multiple datasets, the distilled student model achieved a quadratic weighted kappa of 0.863 and accuracy of 0.865 on pooled testing, while maintaining comparable performance across held‑out environments. The framework also demonstrated improved calibration (expected calibration error 0.025–0.036) and higher accuracy (up to 0.923) when deferring the least confident 20% of images.

By Sohaib Sulman, Muhammad Atif Saeed, Akhtar Jamil, Alaa Ali Hameed, Roland Aydin
arXiv Machine Learning
Jul 28

Same Predictions, Different Reasons: The Effect of Quantization on Model Explanations

arXiv:2607. 22872v1 Announce Type: new Abstract: Post-training quantization (PTQ) has become a practical solution for deploying deep learning models on resource-constrained edge devices by compressing high-precision floating-point weights into low-precision representations without requiring retraining.

By Kazi Kamruzzaman Rabbi, Md. Zami Al Zunaed Farabe, M. Sohel Rahman
Hugging Face Trending Papers
Jul 24

Same Predictions, Different Reasons: The Effect of Quantization on Model Explanations

Post-training quantization (PTQ) has become a practical solution for deploying deep learning models on resource-constrained edge devices by compressing high-precision floating-point weights into low-precision representations without requiring retraining. Past research has demonstrated that quantization largely preserves classification accuracy; however, whether it also preserves the model's internal reasoning remains an open question.