Gaze Attention is a new mechanism for multimodal large language models that selectively focuses on relevant visual regions during each generation step, rather than attending to all visual tokens. By grouping tokens into spatial regions and using learnable context tokens to retain global information, it reduces attention computation and visual key‑value entries by up to 90%. Experiments on 13 image and 6 video benchmarks show that Gaze Attention matches or outperforms dense‑attention baselines while using fewer visual resources.
By Junha Song, Byeongho Heo, Geonmo Gu, Jaegul Choo, Dongyoon Han, Sangdoo Yun
arXiv:2608.21762v1 Announce Type: cross
Abstract: Vision-language models (VLMs) fail many detail-centric questions for a concrete reason: the answer is visible in the image, yet lost after the image...
By Jinchang Zhu, Rong Fu, Yi Ding, Chenghao Wu, Ying Liu, Menglin Yang
arXiv:2605.26380v2 Announce Type: replace-cross
Abstract: Frontier multimodal large language models (MLLMs) have been reported to achieve over 90\% accuracy on fine-grained perception benchmarks. How...
By Jingru Chen, Yiming Liu, Mingtao Chen, Sijie Chen, Richeng Xuan, Liang Yang, Zhichao Hu, Fanyang Lu
arXiv:2510. 00054v3 Announce Type: replace-cross Abstract: Multimodal Large Language Models (MLLMs) have made significant strides in visual understanding tasks.
By Xianjie Liu, Yiman Hu, Yixiong Zou, Liang Wu, Jian Xu, Bo Zheng
arXiv:2608.30705v1 Announce Type: new
Abstract: Multimodal large language models (MLLMs) struggle with fine-grained Visual Search, the task of locating small or rare objects in high-resolution images...
By Jingyi He, Sanghwan Kim, Zeynep Akata
The paper introduces AttWarp, a lightweight technique that uses a multimodal large language model’s cross‑modal attention to perform rectilinear warping of input images at test time. By reallocating spatial resolution toward query‑relevant regions without altering model weights or architecture, AttWarp preserves global context while making small objects and subtle relationships easier for the model to read. Experiments on five benchmarks and four MLLMs show consistent accuracy gains, improved compositional reasoning, and reduced hallucinations compared to baseline image‑manipulation methods.
By Dwip Dalal, Gautam Vashishtha, Utkarsh Mishra, Jeonghwan Kim, Madhav Kanda, Hyeonjeong Ha, Svetlana Lazebnik, Heng Ji, Unnat Jain
arXiv:2604. 01280v2 Announce Type: replace-cross Abstract: Knowledge-based Visual Question Answering (KB-VQA) requires Multimodal Large Language Models (MLLMs) to identify and combine fine-grained visual cues with retrieved textual evidence.
By Marco Morini, Sara Sarto, Marcella Cornia, Lorenzo Baraldi, Rita Cucchiara
arXiv:2610.02117v1 Announce Type: cross
Abstract: On-policy self-distillation has recently emerged as an effective approach for improving language-model reasoning by supervising students with a froze...
By Sophia Sirko-Galouchenko, Monika Wysoczanska, Andrei Bursuc, Nicolas Thome, Spyros Gidaris
The paper introduces Visual Retrieval Heads (VRHs), a small fraction of attention heads in vision‑language models that are causally responsible for grounding text descriptions to image regions. By recasting head‑scoring methods and evaluating across eleven VLMs and five benchmarks, the authors show that masking the top 20 VRHs can drop grounding accuracy by up to 80 percentage points, while random masking has little effect. VRHs generalize across various visual reference tasks, preserve output format while corrupting localization, and transfer causally across models sharing an LLM backbone.
By Chanho Park, Daehyeon Choi, Jihyun Lee, Minhyuk Sung
The paper investigates how retrieval heads—sparse attention heads that transfer information from context to output—behave when multimodal large language models (MLLMs) are extended to include audio-visual inputs. Across 18 language, vision‑language, and omni‑modal checkpoints, the authors find that retrieval heads remain sparse and largely shared across modalities, with less than 2 % of heads being common to all modalities; ablating these shared heads degrades tasks such as document understanding, spatial perception, and visual mathematics. By training retrieval heads with a ranking objective, they demonstrate a 6.03‑point improvement in Recall@1 on the MMDocIR benchmark for Qwen3‑VL‑8B and improved document retrieval across three model families.
By Jia Wang, Yuqing Li, Jiangnan Li, Zheng Lin, Mo Yu
arXiv:2601.19060v2 Announce Type: replace-cross
Abstract: Multimodal Retrieval-Augmented Generation (MM-RAG) augments visual question answering (VQA) with external knowledge, but many systems rely on...
By Jeonghwan Kim, Renjie Tao, Sanat Sharma, Jiaqi Wang, Kai Sun, Zhaojiang Lin, Seungwhan Moon, Lambert Mathias, Anuj Kumar, Heng Ji, Xin Luna Dong
The paper demonstrates that vision‑language models (VLMs) possess a small set of attention heads, called Visual Retrieval Heads (VRHs), that are causally responsible for linking text prompts to specific image regions. By adapting head‑scoring techniques from language models, the authors identify VRHs as the heads whose attention from output prediction tokens, summed over the ground‑truth referent region, most reliably indicates causal grounding. Experiments across eleven VLMs and five referring‑expression benchmarks show that masking the top 20 VRHs can drop grounding accuracy by up to 80 percentage points, while random masking has little effect, and that VRHs generalize across diverse visual tasks and transfer across models sharing an LLM backbone.