arXiv Computer Vision

Zoom Where the Answer Looks: Cross-Resolution Self-Distillation for MLLMs under Visual-Token Budgets

arXiv Computer Vision
Sep 18

Region-Level Policy Optimization for Fine-grained MLLM Perception

The paper introduces Vision‑RL2, a region‑level reinforcement learning approach that optimizes a lightweight proposal network for fine‑grained multimodal large language model (MLLM) perception. By treating coherent image regions as actions and scoring them with a frozen MLLM reader, the method selectively focuses visual resolution on evidence, reducing token usage while improving accuracy across multiple benchmarks and backbones. The approach eliminates the need for region annotations, response sampling, or reasoning trajectories, and the refined proposals enable sparse encoding that magnifies relevant evidence.

By Yuheng Shi, Xiaohuan Pei, Minjing Dong, Chang Xu
arXiv AI
6d ago

ZoomGain: Mining Global-to-Zoom Perception Gaps for Fine-Grained Visual Supervision

ZoomGain introduces a data construction pipeline that exploits the global-to-zoom perception gap to generate fine‑grained visual questions. By creating questions from full‑image context and verifying answers only on zoomed‑in views, the method ensures unambiguous, language‑prior‑free queries and high‑quality distractors for multiple‑choice tasks. Using this pipeline, the authors built ZoomGainBench, a 1,000‑question bilingual benchmark, and demonstrated that training on 9 K ZoomGain samples markedly boosts the visual perception abilities of 4B/9B VLMs, surpassing many larger models on perception benchmarks.

By Haozhan Shen, Liangbo He, Jun Lan, Huijia Zhu, Weiqiang Wang, Tiancheng Zhao, Jianwei Yin
arXiv Computer Vision
Aug 28

Retrieval Heads Meet Vision: Uncovering How VLMs Locate and Extract Visual Information

The paper introduces Visual Retrieval Heads (VRHs), a small fraction of attention heads in vision‑language models that are causally responsible for grounding text descriptions to image regions. By recasting head‑scoring methods and evaluating across eleven VLMs and five benchmarks, the authors show that masking the top 20 VRHs can drop grounding accuracy by up to 80 percentage points, while random masking has little effect. VRHs generalize across various visual reference tasks, preserve output format while corrupting localization, and transfer causally across models sharing an LLM backbone.

By Chanho Park, Daehyeon Choi, Jihyun Lee, Minhyuk Sung
arXiv Computer Vision
Oct 5

Gaze Attention: Query-Adaptive Visual Routing for Efficient Multimodal LLMs

Gaze Attention is a new mechanism for multimodal large language models that selectively focuses on relevant visual regions during each generation step, rather than attending to all visual tokens. By grouping tokens into spatial regions and using learnable context tokens to retain global information, it reduces attention computation and visual key‑value entries by up to 90%. Experiments on 13 image and 6 video benchmarks show that Gaze Attention matches or outperforms dense‑attention baselines while using fewer visual resources.

By Junha Song, Byeongho Heo, Geonmo Gu, Jaegul Choo, Dongyoon Han, Sangdoo Yun
arXiv Machine Learning
Sep 30

Constructive Distortion: Improving MLLMs with Attention-Guided Image Warping

The paper introduces AttWarp, a lightweight technique that uses a multimodal large language model’s cross‑modal attention to perform rectilinear warping of input images at test time. By reallocating spatial resolution toward query‑relevant regions without altering model weights or architecture, AttWarp preserves global context while making small objects and subtle relationships easier for the model to read. Experiments on five benchmarks and four MLLMs show consistent accuracy gains, improved compositional reasoning, and reduced hallucinations compared to baseline image‑manipulation methods.

By Dwip Dalal, Gautam Vashishtha, Utkarsh Mishra, Jeonghwan Kim, Madhav Kanda, Hyeonjeong Ha, Svetlana Lazebnik, Heng Ji, Unnat Jain