arXiv Computer Vision
On the Necessity of Attention-FFN Split in Vision Transformers
Read the original on arXiv Computer Vision →The Flow has not summarised this story yet — read it at arXiv Computer Vision.
Related stories
arXiv Computation and Language
Aug 31
Semantic Head Specialization Guides Hybrid ViT Attention for Multimodal LLMs
The paper introduces Semantic Head Specialization (SHS), a phenomenon where Vision Transformer (ViT) attention heads specialize as either object- or background-focused, most evident under full attention. It proposes the SHS-Index to quantify this specialization, demonstrating its ability to distinguish full-attention from chunk-window ViTs and its strong correlation with downstream benchmark performance. Leveraging insights into window interaction, token serialization, and local softmax allocation, the authors design Ariadne Attention, a hybrid attention mechanism that matches full-attention performance on 22 image and video tasks while reducing attention compute by 6.5×.
arXiv Computer Vision
Aug 24
Beyond Isolated Heads: Multi-Overlapped-Head Self-Attention for Vision Transformers
arXiv:2410.14874v3 Announce Type: replace Abstract: Multi-Head Self-Attention (MHSA) is the cornerstone of Vision Transformers, allowing models to capture diverse feature representations by projectin...
arXiv AI
Sep 2
HiLRP: Toward One Trustworthy Explanation for Vision Transformer: Conservation-Valid Attribution via Attention Primitives
HiLRP introduces a unified attribution framework for Vision Transformers (ViTs) that addresses the challenges posed by diverse architectural designs. By decomposing ViT operations into four basic types—linear maps, bilinear mixing, normalization/gating, and reindexing—HiLRP applies conservation‑satisfying relevance rules, enabling reliable explanations across a wide range of backbones. The method outperforms 14 existing attribution techniques on 10 architectures, maintaining conservation and improving localization accuracy (0.97 Pointing) compared to competitors.
arXiv Machine Learning
Jun 5
Exact Linear Attention
arXiv:2605. 18848v3 Announce Type: replace Abstract: This paper introduces Exact Linear Attention (ELA), a mechanism that achieves linear computational complexity for Transformer attention by exploiting the exact decomposition property of kernel functions, thereby eliminating approximation error.
arXiv Machine Learning
Jun 4
Vision Transformer Finetuning Benefits from Non-Smooth Components
arXiv:2602. 06883v3 Announce Type: replace Abstract: The smoothness of the transformer architecture has been extensively studied in the context of generalization, training stability, and adversarial robustness.
arXiv Computer Vision
Sep 16
CFGPNet: Cross-Attention-Based Fused Gradient Programmed Network Framework for Multispectral Object Detection
arXiv:2608.06205v2 Announce Type: replace Abstract: Multispectral object detection combines visible and thermal imagery to improve perception under challenging illumination and environmental conditio...