arXiv AI
Jun 19
The Hidden Evolution of Disguised Visual Context inside the VLM
arXiv:2606. 20077v1 Announce Type: cross Abstract: Visual tokens enter Large Language Models (LLMs) as raw, foreign signals.
GeoPID: Decomposing and Steering Visual Information in Vision-Language Models
Related stories
arXiv Machine Learning
Jun 8
The Dual Mechanisms of Spatial Variable Binding in Vision-Language Models
arXiv:2603. 22278v2 Announce Type: replace-cross Abstract: Many multimodal tasks, such as image captioning and visual question answering, require vision-language models (VLMs) to bind objects with their properties and spatial relations.
arXiv Computer Vision
Sep 24
VIVAS: Vitalizing Visual Perception in VLM Pre-training via Vision-language Unified Autoregressive Supervision
VIVAS is a new Vision‑Language Model pre‑training framework that addresses the lack of fine‑grained visual perception in existing VLMs. It introduces a unified token space and a dense‑structural‑semantic vision tokenizer that expands the textual vocabulary with visual tokens, enabling vision‑language unified autoregressive supervision over both visual details and linguistic content. Trained on 12.4 T tokens, VIVAS achieves state‑of‑the‑art results on 7 tasks and 39 multimodal benchmarks.
arXiv AI
Aug 6
Chain-of-Visual-Thought: Teaching VLMs to See and Think Better with Continuous Visual Tokens
arXiv:2511. 19418v3 Announce Type: replace-cross Abstract: Vision-Language Models (VLMs) excel at reasoning in linguistic space but struggle with perceptual understanding that requires dense visual perception, e.
arXiv AI
6d ago
Where-OPD: Spatially Guided On-Policy Self-Distillation of MLLMs with Synthetic Scenes
arXiv:2610.02117v1 Announce Type: cross Abstract: On-policy self-distillation has recently emerged as an effective approach for improving language-model reasoning by supervising students with a froze...
arXiv Computer Vision
Sep 24
UVU: Improving Multimodal Understanding via Vision-Language Unified Autoregressive Paradigm
UVU is a vision-language unified autoregressive framework that integrates visual supervision directly into the pre-training stage of multimodal large language models. By using continuous visual encoding and a large-scale iterative hierarchical clustering algorithm to build a pixel-level visual codebook, UVU enables lossless representation of visual inputs and autoregressive generation of pixel-level image tokens alongside textual tokens. This approach synergizes pixel-level visual perception with semantic-level visual understanding, allowing models to internalize visual reconstruction capabilities and improve multimodal understanding performance.
Hugging Face Trending Papers
Aug 3
Mitigating Visual Degradation in MLLMs via Spatial-Spectral Visual Anchor Learning
Despite the progress of multimodal large language models (MLLMs), they continue to exhibit deficiencies in visual perception. Following visual instruction tuning, internal MLLM representations rapidly deviate from their original semantic states during inference, causing severe information degradation.
arXiv AI
Jun 2
CLIP Tricks You: Training-free Token Pruning for Efficient Pixel Grounding in Large VIsion-Language Models
arXiv:2605. 13178v2 Announce Type: replace-cross Abstract: In large vision-language models, visual tokens typically constitute the majority of input tokens, leading to substantial computational overhead.
arXiv AI
Oct 1
GaugeVLM: Structuring Spatial Supervision with Measured Geometric Interventions
arXiv:2609.38285v1 Announce Type: cross Abstract: Vision-language models (VLMs) can contradict themselves across views of the same spatial relation and fail to respond when that relation changes. Add...
arXiv Computer Vision
Sep 3
TempoGround: State-Aware Streaming Visual Grounding with Vision-Language Models
TempoGround is a vision‑language model–native framework for streaming visual grounding that detects cross‑frame object correspondence and explicitly models object presence states. It uses a curriculum prediction mechanism to resolve 2D instance association, predict object entry, continuation, or exit, decode 2D boxes, and lift them to 3D camera‑frame boxes. The approach is further refined with Streaming Grounding Reinforcement, which optimizes grounding, identity, and consistency rewards, and achieves significant improvements on multiple streaming visual grounding benchmarks.
arXiv Computer Vision
Aug 25
ViSMoE: Visual-Aware Sparse Mixture-of-Experts for Embodied Referring Expression Grounding
ViSMoE introduces a visual‑aware sparse Mixture‑of‑Experts framework for embodied referring expression grounding, enabling an agent to navigate real environments and localize a target object from natural language instructions. By routing visual information through specialized experts, the method produces discriminative representations for both navigation views and candidate objects, unlike prior approaches that use a single vision encoder. Experiments on the REVERIE and SOON datasets show that ViSMoE surpasses existing state‑of‑the‑art methods.
arXiv Computer Vision
Sep 3
Beyond Appearance: Can Multimodal Large Language Models Exploit Vertical Structure for Remote Sensing Natural Scene Understanding?
The paper introduces VertiCue-Bench, a diagnostic benchmark designed to test whether multimodal large language models (MLLMs) can perceive, ground, and utilize vertical structure information in remote-sensing natural scenes. It presents a three-stage framework—Perception, Grounding, Utilization—and a Representation Intervention Spectrum across various modalities to evaluate ten state-of-the-art models. The study finds a significant Vertical Structure Utilization Gap: while models show some geometric perception, they struggle to accurately link vertical evidence to spatial entities and incorporate it into semantic decisions.