PailitaoGR is a generative image retrieval method that incorporates target-focused perception and selective auxiliary-evidence utilization. It uses a target Enhancer and on-policy distillation to highlight search-target regions, and an auxiliary enhancer with incremental contrastive distillation to exploit auxiliary evidence. Trained on real-world online image-search logs, it achieves an average 13.8% improvement over existing baselines.
The paper introduces Visual Retrieval Heads (VRHs), a small fraction of attention heads in vision‑language models that are causally responsible for grounding text descriptions to image regions. By recasting head‑scoring methods and evaluating across eleven VLMs and five benchmarks, the authors show that masking the top 20 VRHs can drop grounding accuracy by up to 80 percentage points, while random masking has little effect. VRHs generalize across various visual reference tasks, preserve output format while corrupting localization, and transfer causally across models sharing an LLM backbone.
By Chanho Park, Daehyeon Choi, Jihyun Lee, Minhyuk Sung
arXiv:2601.19060v2 Announce Type: replace-cross
Abstract: Multimodal Retrieval-Augmented Generation (MM-RAG) augments visual question answering (VQA) with external knowledge, but many systems rely on...
By Jeonghwan Kim, Renjie Tao, Sanat Sharma, Jiaqi Wang, Kai Sun, Zhaojiang Lin, Seungwhan Moon, Lambert Mathias, Anuj Kumar, Heng Ji, Xin Luna Dong
The paper demonstrates that vision‑language models (VLMs) possess a small set of attention heads, called Visual Retrieval Heads (VRHs), that are causally responsible for linking text prompts to specific image regions. By adapting head‑scoring techniques from language models, the authors identify VRHs as the heads whose attention from output prediction tokens, summed over the ground‑truth referent region, most reliably indicates causal grounding. Experiments across eleven VLMs and five referring‑expression benchmarks show that masking the top 20 VRHs can drop grounding accuracy by up to 80 percentage points, while random masking has little effect, and that VRHs generalize across diverse visual tasks and transfer across models sharing an LLM backbone.
arXiv:2610.28382v1 Announce Type: new
Abstract: Training multimodal search agents requires tasks that make visually grounded search necessary for determining the answer. Textual identity clues and un...
By Eric Tao, Minhao Sun, Liyan Liu, Zhijing Wu, Xiaohan Yu, Lang Mei, Li Huang, Chao Feng, Jinchao Ma, Sichen Kang, Siyu Mo, Chong Chen
arXiv:2604. 01280v2 Announce Type: replace-cross Abstract: Knowledge-based Visual Question Answering (KB-VQA) requires Multimodal Large Language Models (MLLMs) to identify and combine fine-grained visual cues with retrieved textual evidence.
By Marco Morini, Sara Sarto, Marcella Cornia, Lorenzo Baraldi, Rita Cucchiara