arXiv AI

HiLRP: Toward One Trustworthy Explanation for Vision Transformer: Conservation-Valid Attribution via Attention Primitives

HiLRP introduces a unified attribution framework for Vision Transformers (ViTs) that addresses the challenges posed by diverse architectural designs. By decomposing ViT operations into four basic types—linear maps, bilinear mixing, normalization/gating, and reindexing—HiLRP applies conservation‑satisfying relevance rules, enabling reliable explanations across a wide range of backbones. The method outperforms 14 existing attribution techniques on 10 architectures, maintaining conservation and improving localization accuracy (0.97 Pointing) compared to competitors.

arXiv AI

Sep 16

ResLRP: The Role of Residual Cancellation in Attribution Instability in Vision Transformers

The paper introduces ResLRP, an extension of Layer-wise Relevance Propagation that explicitly handles residual connections in Vision Transformers to prevent attribution explosion. It demonstrates that residual cancellation causes instability in ViT explanations, and that ResLRP improves faithfulness and localization across a wide range of ViT architectures, including Vision Language Models. The method also provides a diagnostic measure for predicting attribution degradation and successfully localizes Sparse Autoencoder features.

By Jim Berend, Reduan Achtibat, Daniel Sch\"affer, Alexander Binder, Wojciech Samek, Sebastian Lapuschkin, Maximilian Dreyer

llms multimodal benchmarks

arXiv Machine Learning

Jun 16

Spatial Priors via Space Filling Curves for Small and Limited Data Vision Transformers

arXiv:2606. 14757v1 Announce Type: cross Abstract: Though Vision Transformers (ViTs) have become the dominant backbone in many computer vision tasks, due to permutation equivariance, their attention mechanism lacks explicit spatial inductive biases.

By Leyla Naz Candogan, Arshia Afzal, Pol Puigdemont, Volkan Cevher

llms computer-vision fine-tuning safety

arXiv AI

4d ago

Attention Sinks in VLMs Are Borrowed, Not Seen

arXiv:2610.15855v1 Announce Type: new Abstract: We study how to detect and mitigate visual attention sinks in vision-language models (VLMs). Visual attention sinks are image tokens that absorb a larg...

By Dengjia Zhang, Alexander Martin, Pengyu Yan, Xihan Sun, Jiaqi Li, Benjamin Van Durme, Andrew Yates, Reno Kriz, William Walden

llms multimodal benchmarks

arXiv Computer Vision

Sep 1

Fast and Accurate Image Restoration and Generation with Rank Enhanced Linear Attention

arXiv:2505.16157v3 Announce Type: replace Abstract: Transformer-based models have made remarkable progress in image restoration (IR) tasks. However, the quadratic complexity of self-attention in Tran...

By Yuang Ai

llms diffusion computer-vision benchmarks

arXiv Computer Vision

Oct 2

Subtoken Vision Transformer for Fine-grained Recognition

arXiv:2607.09086v2 Announce Type: replace Abstract: We present Subtoken Vision Transformer (SubViT), a selective image tokenization method for fine-grained visual recognition. Standard Vision Transfo...

By Jie Zhu, Ivy Zhang, Minchul Kim, Xiaoming Liu

llms computer-vision nlp

arXiv AI

Jun 9

RAPID: Layer-Wise Redundancy-Aware Pruning and Importance-Driven Token Merging for Efficient ViT

arXiv:2606. 08156v1 Announce Type: cross Abstract: Vision Transformers (ViTs) achieve strong performance but suffer from high computational costs due to quadratic self-attention complexity.

By Kyumin Choi, Ikbeom Jang

llms efficiency

arXiv Machine Learning

Jun 5

Exact Linear Attention

arXiv:2605. 18848v3 Announce Type: replace Abstract: This paper introduces Exact Linear Attention (ELA), a mechanism that achieves linear computational complexity for Transformer attention by exploiting the exact decomposition property of kernel functions, thereby eliminating approximation error.

By Weinuo Ou

llms computer-vision reinforcement-learning efficiency safety

arXiv AI

Jun 30

ScAle: Attention Head Scaling as a Minimal Adapter for Spatial Reasoning in Vision Language Models

arXiv:2606. 29579v1 Announce Type: cross Abstract: Spatial reasoning remains a persistent challenge for many vision language models (VLMs), and improving it typically requires fine-tuning with substantial additional parameters.

By Rahul Chowdhury, Timothy A Rupprecht, Xuan Shen, Pu Zhao, Yanzhi Wang

llms fine-tuning benchmarks

arXiv Machine Learning

Sep 10

Learn to Rank: Visual Attribution by Learning Importance Ranking

arXiv:2604.05819v2 Announce Type: replace-cross Abstract: Interpreting the decisions of complex computer vision models is crucial to establish trust and accountability, especially in safety-critical...

By David Schinagl, Christian Fruhwirth-Reisinger, Alexander Prutsch, Samuel Schulter, Horst Possegger

llms computer-vision safety

arXiv Computer Vision

Oct 2

Do Vision Transformers Need All-to-All Attention? Global Communication Through Elastic Learned Cores

arXiv:2605.12491v2 Announce Type: replace Abstract: Vision Transformers (ViTs) learn rich visual-semantic representations through all-to-all self-attention among patch tokens. However, this design im...

By Alan Z. Song, Yinjie Chen, Mu Nan, Deva Ramanan, Michael J. Tarr, Andrew F. Luo

llms computer-vision benchmarks

arXiv Machine Learning

Oct 9

Dissecting Representation Structure in Vision Transformers: A Rigorous Architectural Study

The paper presents a detailed study of feature representations in Vision Transformers (ViTs), revealing that feature collapse at initialization causes redundancy and proposing a reduction scheme to address it. It demonstrates that entropy and minimum eigenvalue metrics reliably predict generalization, and that token‑space features and linear submodule outputs are particularly informative. The authors develop a proxy that improves ranking of ViT architectures by 18‑48% over previous baselines, enabling the selection of models with higher accuracy at comparable or lower computational cost.

By Kim-Cuc Nguyen, Ngai-Man Cheung

llms rag computer-vision

arXiv Computer Vision

Oct 9

LearnPruner: Rethinking Attention-based Token Pruning in Vision Language Models

LearnPruner rethinks token pruning in vision‑language models by addressing shortcomings of attention mechanisms in vision encoders and large language models. The method uses a two‑stage approach: first a learnable pruning module removes redundant vision tokens after the vision encoder, then only task‑relevant tokens are kept in the LLM’s middle layer. Experiments show that LearnPruner retains about 95% of performance while using only 5.5% of vision tokens, achieving a 3.2× speed‑up.

By Rinyoichi Takezoe, Yaqian Li, Zihao Bo, Anzhou Hou, Mo Guang, Kaiwen Long

llms efficiency multimodal safety

Related stories