HiLRP introduces a unified attribution framework for Vision Transformers (ViTs) that addresses the challenges posed by diverse architectural designs. By decomposing ViT operations into four basic types—linear maps, bilinear mixing, normalization/gating, and reindexing—HiLRP applies conservation‑satisfying relevance rules, enabling reliable explanations across a wide range of backbones. The method outperforms 14 existing attribution techniques on 10 architectures, maintaining conservation and improving localization accuracy (0.97 Pointing) compared to competitors.
By Sathiyamohan Nishankar, Pubudu Sanjeewani, Asanka Perera, Selvarajah Thuseethan
The paper introduces SAGE, a lightweight method that guides decoder attention in vision-language models away from prompt‑invariant attention sinks (PIS) toward query‑dependent regions of interest (ROIs). By using token‑aligned ROI masks from vision backbones like CLIP, ViT, and DINOv3, SAGE improves visual grounding and reduces hallucinations across various VLM families. Experiments show consistent gains on public vision‑language benchmarks, especially in fine‑grained visual discrimination tasks where localized evidence is essential.
By Jeonghyo Song, YoungJoon Yoo
arXiv:2509. 22415v3 Announce Type: replace-cross Abstract: Multimodal large language models (MLLMs) have achieved strong vision-language performance, yet their token-level visual evidence remains difficult to inspect.
By Jiawei Liang, Jianjie Huang, Ruoyu Chen, Xianghao Jiao, Siyuan Liang, Shiming Liu, Xiaochun Cao
arXiv:2507. 18043v2 Announce Type: replace-cross Abstract: Inference-time steering methods offer a lightweight alternative to fine-tuning large language models (LLMs) and vision-language models (VLMs) by modifying internal activations at test time without updating model weights.
By Duy Nguyen, Archiki Prasad, Elias Stengel-Eskin, Mohit Bansal
The paper introduces Visual Retrieval Heads (VRHs), a small fraction of attention heads in vision‑language models that are causally responsible for grounding text descriptions to image regions. By recasting head‑scoring methods and evaluating across eleven VLMs and five benchmarks, the authors show that masking the top 20 VRHs can drop grounding accuracy by up to 80 percentage points, while random masking has little effect. VRHs generalize across various visual reference tasks, preserve output format while corrupting localization, and transfer causally across models sharing an LLM backbone.
By Chanho Park, Daehyeon Choi, Jihyun Lee, Minhyuk Sung
arXiv:2607. 10365v1 Announce Type: cross Abstract: Vision Transformers (ViTs) are difficult to interpret because current methods of relevance propagation and attention flow do not fully consider some key architectural features, such as the uneven importance of attention heads and residual connections.
By Christopher Buratti, Michele Marchetti, Federica Parlapiano, Davide Traini, Domenico Ursino, Luca Virgili
arXiv:2606. 29431v1 Announce Type: new Abstract: Despite the impressive capabilities of Large Vision-Language Models (LVLMs), they remain susceptible to hallucination, generating content inconsistent with the input image.
By Yichen Guo, Kai Tang, Fenglai Lin, Yiding Sun, Dongshuo Zhang, Wenya Wang, Lin William Cong, Shanghang Zhang
arXiv:2603. 24058v2 Announce Type: replace-cross Abstract: Object hallucination in Large Vision-Language Models (LVLMs) severely compromises their reliability in real-world applications, posing a critical barrier to their deployment in high-stakes scenarios such as autonomous driving and medical image analysis.
By Han Sun, Qin Li, Peixin Wang, Min Zhang
The paper investigates attention sinks—tokens or positions that attract disproportionate attention—in text-to-image transformers, comparing bidirectional diffusion models and causal autoregressive models. It finds that both model types exhibit attention sinks, but with distinct patterns: autoregressive models focus on the first and global tokens, while diffusion models develop sinks near modality boundaries, within text regions, and at spatial corners. Perturbation experiments show that sink-related heads influence generation more than non-sink heads, and introducing register tokens can redistribute attention without harming performance.
By Binglei Li, Mengping Yang, Zhiyu Tan, Wenxu Qian, Junping Zhang, Hao Li
arXiv:2609.07876v1 Announce Type: cross
Abstract: Recent methods in language model interpretability employ techniques such as sparse autoencoders to decompose residual stream contributions into linea...
By Arjun Patrawala, Jiahai Feng, Erik Jones, Jacob Steinhardt
arXiv:2606. 06890v1 Announce Type: cross Abstract: Vision-Language Models (VLMs) frequently rely on language priors, producing confident answers that are weakly grounded in visual evidence.
By Runyu Zhou, Qi Zhang, Qixun Wang, Yisen Wang
FusionAudit introduces PaRMA, a pathway-conditioned robustness auditing framework for native multimodal vision‑language models. It evaluates two image‑level interventions—pixel perturbation and rendered‑text insertion—to expose architecture‑specific blind spots. The study applies diagnostic‑motivated attacks (MANTIS and Nat‑Evid) to 18 open‑weight VLMs, revealing that legacy single‑attack rankings are unstable and that PaRMA uncovers higher residual risk than standard benchmarks.
By Jiaming Zhang, Zherui Li, Siqi Guo, Junhao Dong, Kun Wang, Wei Yang Bryan Lim