arXiv Machine Learning
Attention Transfer Is Not Universally Effective for Vision Transformers
Read the original on arXiv Machine Learning →The Flow has not summarised this story yet — read it at arXiv Machine Learning.
Related stories
arXiv AI
Jul 22
A Controlled Study of Attention-Only Transformers
arXiv:2607. 18363v1 Announce Type: cross Abstract: Feed-forward networks hold two thirds of a transformer's non-embedding parameters, yet the architecture has not received a necessity test that controls parameters, compute, and depth at once.
arXiv Machine Learning
Jun 16
Taylor-Calibrate: Principled Initialization for Hybrid Linear Attention Distillation
arXiv:2606. 16429v1 Announce Type: new Abstract: Hybrid linear attention models offer an appealing path to faster long-context inference: they reduce the quadratic cost and KV-cache burden of full softmax attention while retaining much of the quality of Transformer models.
arXiv Machine Learning
Jun 4
Vision Transformer Finetuning Benefits from Non-Smooth Components
arXiv:2602. 06883v3 Announce Type: replace Abstract: The smoothness of the transformer architecture has been extensively studied in the context of generalization, training stability, and adversarial robustness.
arXiv Computer Vision
2d ago
CODI: Improving Self-Supervised Vision Transformers with Cross Distillation
The paper introduces CODI, a method that enhances self‑supervised Vision Transformers by mixing a small portion of the teacher’s class‑token distribution into each teacher patch‑token target. This cross distillation approach gives every masked patch an attention‑independent semantic correction without altering the architecture. Experiments demonstrate that CODI improves performance on both dense prediction and image‑level tasks under aligned pretraining settings.
Hugging Face Trending Papers
3d ago
CODI: Improving Self-Supervised Vision Transformers with Cross Distillation
Self-supervised Vision Transformers learn strong image-level representations, but their patch-level representations can be unreliable for dense prediction. We study this mismatch and find that the cla...
arXiv Computer Vision
Oct 2
Subtoken Vision Transformer for Fine-grained Recognition
arXiv:2607.09086v2 Announce Type: replace Abstract: We present Subtoken Vision Transformer (SubViT), a selective image tokenization method for fine-grained visual recognition. Standard Vision Transfo...