← Back to all news
arXiv Computer Vision By Kabir Thayani

Breaking the Geometric Bottleneck: Contrastive Expansion in Asymmetric Cross-Modal Distillation

Read the original on arXiv Computer Vision →

The paper investigates how knowledge distillation from Vision Transformers to smaller CNNs can cause dimensional collapse in the student’s representation space. Using SVD and Shannon entropy, the authors show that cosine‑based distillation leads to a drastic reduction in effective rank, while adding an InfoNCE objective can double the rank but harms downstream accuracy due to signal dilution. They further demonstrate that a label‑aware contrastive objective (Supervised Contrastive distillation) can maintain or improve accuracy without unnecessary rank expansion, indicating that effective rank alone is not a reliable indicator of representation quality.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computer Vision.

Related stories

arXiv Machine Learning
Oct 12

CV-OPD: Contrastive Vision On-policy Distillation

The paper introduces Contrastive Vision On‑policy Distillation (CV‑OPD), a method that enhances on‑policy distillation for vision‑language models by adding a contrastive divergence loss and a discrepancy‑aware gating mechanism. CV‑OPD contrasts teacher distributions from positive and negative visual crops, forcing the student to rely on genuine visual evidence rather than shortcuts. Experiments on fine‑grained visual benchmarks show that CV‑OPD reduces shortcut learning and improves visual reasoning across existing VLMs.

By Haowei Wang, Hao Lin, Qi Zhu, Jiayi Ji, Deyi Ji, Liujuan Cao, Rongrong Ji
efficiencymultimodalbenchmarks
More like this →
Hugging Face Trending Papers
Aug 11

Grid-Preserving Knowledge Distillation: Transferring Convolutional Inductive Bias to Vision Transformers under Data Scarcity

Vision Transformers underperform convolutional networks when training data is scarce, and distilling convolutional inductive biases from a CNN teacher is an effective remedy that leaves the deployed model unchanged. General-purpose feature distillation, however, transfers little in this setting.

llmsefficiencybenchmarkssafety
More like this →
Hugging Face Trending Papers
Jul 2

DRDN: Decoupled Representation Dynamic Network for From-Scratch ViT Class-Incremental Learning

Dynamic expansion methods for class-incremental learning (CIL) protect task-specific knowledge by growing dedicated tokens or subnetworks, yet our analyses suggest that classification supervision alone does not sufficiently preserve task-agnostic shared backbone representations over long incremental sequences. We identify two intertwined challenges: cross-task confusion from sequential training on predominantly current-task data, which biases decision boundaries toward recent tasks; and under-optimized shared representations in the backbone that cap long-term discriminability as tasks accumulate.

llms
More like this →