arXiv:2610.08401v1 Announce Type: cross Abstract: While recent vision-language models (VLMs) have shown outstanding performance across diverse applications, they tend to under-use visual information...
The paper introduces Visual Retrieval Heads (VRHs), a small fraction of attention heads in vision‑language models that are causally responsible for grounding text descriptions to image regions. By recasting head‑scoring methods and evaluating across eleven VLMs and five benchmarks, the authors show that masking the top 20 VRHs can drop grounding accuracy by up to 80 percentage points, while random masking has little effect. VRHs generalize across various visual reference tasks, preserve output format while corrupting localization, and transfer causally across models sharing an LLM backbone.
arXiv:2501.04001v4 Announce Type: replace Abstract: This work presents Sa2VA, the first comprehensive, unified model for dense grounded understanding of both images and videos. Unlike existing multi-...
arXiv:2609.24510v2 Announce Type: replace Abstract: Recent advances in vision foundation models (VFMs) have shown remarkable capabilities across diverse unimodal visual tasks. However, adapting VFMs...
Abundant visual information strengthens vision-language model (VLM) perception, yet massive visual tokens raise inference costs. Existing visual token pruning methods rely on similarity-based guidance, which exploits pairwise text-vision and vision-vision token correlations for compression.
arXiv:2608. 07886v1 Announce Type: cross Abstract: Vision-language grounding connects language to visual content, yet most existing formulations reduce grounding to a unidirectional localization problem: given a prespecified text phrase or category name, identify the corresponding image region.
arXiv:2502.04192v5 Announce Type: replace Abstract: Multiple works have emerged to push the boundaries of multi-modal large language models (MLLMs) towards pixel-level understanding. The current tren...
VIVAS is a new Vision‑Language Model pre‑training framework that addresses the lack of fine‑grained visual perception in existing VLMs. It introduces a unified token space and a dense‑structural‑semantic vision tokenizer that expands the textual vocabulary with visual tokens, enabling vision‑language unified autoregressive supervision over both visual details and linguistic content. Trained on 12.4 T tokens, VIVAS achieves state‑of‑the‑art results on 7 tasks and 39 multimodal benchmarks.
The paper investigates whether frozen Multimodal Large Language Models (MLLMs) encode grounding information about specific visual instances within the tokens they generate, including modifiers such as attributes, spatial expressions, and relational terms. The authors introduce OTTER, a lightweight supervised probe that applies Optimal Transport to align generated tokens with visual regions, producing compact grounding maps. Experiments demonstrate that modifier tokens, especially spatial terms, carry instance‑discriminative visual information, that object nouns also encode referential cues, and that the recovered grounding remains robust to context changes and generalizes to free generation and cross‑dataset transfer.
arXiv:2606. 15134v1 Announce Type: cross Abstract: Vision encoders for retrieval are typically trained with class-label supervision: each training pair reduces to a scalar that uniformly pushes the embedding apart or pulls it together, as if every visual attribute either differed or matched.
The paper introduces O$^2$-VG, a unified framework for oriented object visual grounding in remote sensing images, comprising three complementary models: O$^2$-VG-Trans, a cross‑modality transformer; O$^2$-VG-Uni, which predicts universal oriented proposals; and O$^2$-VG-VLM, an autoregressive vision‑language model that generates oriented bounding boxes. It also presents DIOR‑R‑SVG, a new dataset containing image, expression, and oriented box triplets for training and evaluation. The framework demonstrates superior performance across multiple benchmarks and is supported by publicly available code.
CatchUp is a training‑free framework that improves efficiency in large vision‑language models by depth‑aware re‑admission of visual tokens. It repeatedly evaluates query‑conditioned relevance, processes only selected tokens per decoder interval, and preserves deferred tokens for later re‑admission, using reference‑to‑current transition signals to approximate missed representation evolution. Experiments on three referring‑expression grounding datasets and six multimodal benchmarks show that CatchUp retains over 92% of full‑model performance with 88.9% fewer active tokens and reduces end‑to‑end latency by 8.0% at a 64‑token budget.