The paper introduces ResLRP, an extension of Layer-wise Relevance Propagation that explicitly handles residual connections in Vision Transformers to prevent attribution explosion. It demonstrates that residual cancellation causes instability in ViT explanations, and that ResLRP improves faithfulness and localization across a wide range of ViT architectures, including Vision Language Models. The method also provides a diagnostic measure for predicting attribution degradation and successfully localizes Sparse Autoencoder features.
By Jim Berend, Reduan Achtibat, Daniel Sch\"affer, Alexander Binder, Wojciech Samek, Sebastian Lapuschkin, Maximilian Dreyer
HiLRP introduces a unified attribution framework for Vision Transformers (ViTs) that addresses the challenges posed by diverse architectural designs. By decomposing ViT operations into four basic types—linear maps, bilinear mixing, normalization/gating, and reindexing—HiLRP applies conservation‑satisfying relevance rules, enabling reliable explanations across a wide range of backbones. The method outperforms 14 existing attribution techniques on 10 architectures, maintaining conservation and improving localization accuracy (0.97 Pointing) compared to competitors.
By Sathiyamohan Nishankar, Pubudu Sanjeewani, Asanka Perera, Selvarajah Thuseethan
arXiv:2508.08966v2 Announce Type: replace
Abstract: The attention mechanism lies at the core of the transformer architecture, providing an interpretable model-internal signal that has motivated a gro...
By Marte Eggen, Jacob Lysn{\ae}s-Larsen, Inga Str\"umke
arXiv:2605. 18848v3 Announce Type: replace Abstract: This paper introduces Exact Linear Attention (ELA), a mechanism that achieves linear computational complexity for Transformer attention by exploiting the exact decomposition property of kernel functions, thereby eliminating approximation error.
By Weinuo Ou
arXiv:2606. 23763v1 Announce Type: cross Abstract: Recent work typically assesses vision--language consistency using attention distributions of answer-side tokens.
By Yiyang Chen, Yixin Tan, Binrui Shen
arXiv:2609.37581v1 Announce Type: cross
Abstract: Vision-Language Models (VLMs) excel at visual understanding and reasoning but often incur substantial inference costs due to the large number of visu...
By Jing Wang, Zhiping Wu, Dongdong Ren, Youfang Han, Wei Zhao, Wenbin Li
The paper introduces Visual Retrieval Heads (VRHs), a small fraction of attention heads in vision‑language models that are causally responsible for grounding text descriptions to image regions. By recasting head‑scoring methods and evaluating across eleven VLMs and five benchmarks, the authors show that masking the top 20 VRHs can drop grounding accuracy by up to 80 percentage points, while random masking has little effect. VRHs generalize across various visual reference tasks, preserve output format while corrupting localization, and transfer causally across models sharing an LLM backbone.
By Chanho Park, Daehyeon Choi, Jihyun Lee, Minhyuk Sung
The paper introduces QK Product Steering, a data‑free, training‑free method that edits the query‑key product in vision‑language models to reduce object hallucination. By suppressing a few dominant singular modes in selected middle layers and mapping the edited product back to query weights, the approach lowers hallucination rates without affecting inference cost. Experiments on three GQA‑based VLMs show a 4.0% average reduction in CHAIR$_s$, with the effect localized to symmetric mutual‑attention channels.
By Karn Tiwari, Varnith Chordia, Prathosh A P
arXiv:2608.23921v1 Announce Type: new
Abstract: Recent Vision-Language Models encode high-resolution images into long visual token sequences, incurring prohibitive prefill costs. To compress them, ex...
By Yuanhao Sun, Huawei Ji, Yuan Jin, Cheng Deng, Luoyi Fu, Xinbing Wang
The paper introduces Semantic Head Specialization (SHS), a phenomenon where Vision Transformer (ViT) attention heads specialize as either object- or background-focused, most evident under full attention. It proposes the SHS-Index to quantify this specialization, demonstrating its ability to distinguish full-attention from chunk-window ViTs and its strong correlation with downstream benchmark performance. Leveraging insights into window interaction, token serialization, and local softmax allocation, the authors design Ariadne Attention, a hybrid attention mechanism that matches full-attention performance on 22 image and video tasks while reducing attention compute by 6.5×.
By Chenhong He, Lei Li, Shicheng Li, Hanglong Lv, Lingpeng Kong, Qi Liu, Tong Yang, Shuhuai Ren
arXiv:2505. 03201v4 Announce Type: replace-cross Abstract: Integrated Gradients (IG) is a widely used attribution method in explainable AI, particularly in computer vision applications where reliable feature attribution is essential.
By Kien Tran Duc Tuan, Tam Nguyen Trong, Son Nguyen Hoang, Khoat Than, Anh Nguyen Duc
arXiv:2606. 14757v1 Announce Type: cross Abstract: Though Vision Transformers (ViTs) have become the dominant backbone in many computer vision tasks, due to permutation equivariance, their attention mechanism lacks explicit spatial inductive biases.
By Leyla Naz Candogan, Arshia Afzal, Pol Puigdemont, Volkan Cevher