arXiv:2609.22588v1 Announce Type: new Abstract: Vision-language models (VLMs) perform strongly on visual question answering benchmarks, yet often make decisions that contradict visual evidence they h...
arXiv:2308. 06035v4 Announce Type: replace Abstract: Humans routinely draw on visual context to predict upcoming words.
arXiv:2608. 16514v1 Announce Type: cross Abstract: Human visual search is serial: the fovea must land on a candidate to confirm it, and those landings form a scanpath.
The study investigates whether attention weights in Vision‑Language Models (VLMs) accurately reflect model reasoning for visual inputs. Using causal perturbation analysis, it identifies three distinct processing modes—Faithful‑Sufficient, Faithful‑Distributed, and Non‑Focal—indicating heterogeneous visual attention faithfulness. The research also shows that human‑annotated ground‑truth regions align with model attention in only about 60% of cases, highlighting a systematic divergence between model visual reliance and human intuition across VQA, document, and chart tasks.
Gaze Attention is a new mechanism for multimodal large language models that selectively focuses on relevant visual regions during each generation step, rather than attending to all visual tokens. By grouping tokens into spatial regions and using learnable context tokens to retain global information, it reduces attention computation and visual key‑value entries by up to 90%. Experiments on 13 image and 6 video benchmarks show that Gaze Attention matches or outperforms dense‑attention baselines while using fewer visual resources.
The study compares human and vision‑language model (VLM) responses to cross‑modal association tasks, using identical stimuli (a pseudo‑word and two images) and recording both choices and eye movements. While larger VLMs show some alignment with human choices, their attention patterns correlate poorly with human gaze, performing no better than a simple center‑bias baseline. Fine‑tuning VLMs on human choices improves choice alignment but not attention alignment, and training on human gaze improves attention correlation without affecting choice accuracy.
The paper introduces SAGE, a lightweight method that guides decoder attention in vision-language models away from prompt‑invariant attention sinks (PIS) toward query‑dependent regions of interest (ROIs). By using token‑aligned ROI masks from vision backbones like CLIP, ViT, and DINOv3, SAGE improves visual grounding and reduces hallucinations across various VLM families. Experiments show consistent gains on public vision‑language benchmarks, especially in fine‑grained visual discrimination tasks where localized evidence is essential.
arXiv:2606. 14703v1 Announce Type: cross Abstract: How a vision-language model internally solves the task of describing an image is far from obvious.
arXiv:2610.32664v1 Announce Type: new Abstract: Multimodal large language models (MLLMs) read images through visual tokens, and under the small per-image budgets that efficient deployment requires, t...
arXiv:2511. 18735v3 Announce Type: replace-cross Abstract: In this work, we define Foresight Intelligence as the capability to anticipate and interpret future events-an ability essential for applications such as autonomous driving, yet largely overlooked by existing research.
The paper introduces EASE, a method that enhances multimodal reinforcement learning with verifiable rewards (RLVR) by adding visual‑evidence process supervision. EASE transforms annotated evidence regions into smoothed visual‑token targets and uses them to guide attention during RL training, but only on high‑reward trajectories. Experiments on Qwen2.5‑VL‑7B, Qwen3‑VL‑4B, and Qwen3‑VL‑8B show that EASE improves average scores over DAPO by 2.5 to 3.1 points across perception, hallucination, visual math, and multimodal reasoning benchmarks, and diagnostics confirm better alignment of visual attention with annotated evidence.
OpenVAM is a new framework for visual attention modeling that combines a dense saliency map with language‑based explanations. It uses a decoupled design: a visual pathway for precise localization and a vision‑language head that generates grounded what/why explanations. The method is trained in three stages to preserve localization while adding language grounding, and a scalable pipeline creates multi‑domain annotations for evaluation.