PailitaoGR is a generative image retrieval model that incorporates a latent think-with-images approach to better handle real‑world query images. It uses a target‑focused perception mechanism—comprising a target enhancer and on‑policy distillation—to highlight the search target, and a selective auxiliary‑evidence mechanism—using an auxiliary enhancer and incremental contrastive distillation—to exploit useful side information. Trained on real‑world online image‑search logs, the method achieves an average 13.8 % improvement over existing baselines.
By Xiaomeng Fan, Yueran Liu, Shengyu Zhou, Chenghan Fu, Wanxian Guan, Feng Li, Chuan Yu, Jian Xu, Bo Zheng
The paper introduces Visual Retrieval Heads (VRHs), a small fraction of attention heads in vision‑language models that are causally responsible for grounding text descriptions to image regions. By recasting head‑scoring methods and evaluating across eleven VLMs and five benchmarks, the authors show that masking the top 20 VRHs can drop grounding accuracy by up to 80 percentage points, while random masking has little effect. VRHs generalize across various visual reference tasks, preserve output format while corrupting localization, and transfer causally across models sharing an LLM backbone.
By Chanho Park, Daehyeon Choi, Jihyun Lee, Minhyuk Sung
arXiv:2604. 01280v2 Announce Type: replace-cross Abstract: Knowledge-based Visual Question Answering (KB-VQA) requires Multimodal Large Language Models (MLLMs) to identify and combine fine-grained visual cues with retrieved textual evidence.
By Marco Morini, Sara Sarto, Marcella Cornia, Lorenzo Baraldi, Rita Cucchiara
The paper demonstrates that vision‑language models (VLMs) possess a small set of attention heads, called Visual Retrieval Heads (VRHs), that are causally responsible for linking text prompts to specific image regions. By adapting head‑scoring techniques from language models, the authors identify VRHs as the heads whose attention from output prediction tokens, summed over the ground‑truth referent region, most reliably indicates causal grounding. Experiments across eleven VLMs and five referring‑expression benchmarks show that masking the top 20 VRHs can drop grounding accuracy by up to 80 percentage points, while random masking has little effect, and that VRHs generalize across diverse visual tasks and transfer across models sharing an LLM backbone.
arXiv:2604. 09508v2 Announce Type: replace-cross Abstract: Visual Retrieval-Augmented Generation (VRAG) empowers Vision-Language Models to retrieve and reason over visually rich documents.
By Yucheng Shen, Jiulong Wu, Jizhou Huang, Dawei Yin, Lingyong Yan, Min Cao
External memory effectively grounds large language models (LLMs) and vision-language models (VLMs)-based question answering (QA) in relevant multimodal evidence. However, existing memory paradigms represent each memory item in raw text and image forms, so retrieval-based systems must pass the retrieved text or images to the generation LLMs/VLMs, resulting in high token consumption and storage pressure, making it unaffordable for resource-constrained applications.