← Back to all news
arXiv AI By Beth Pearson, Michael Wray, Martha Lewis

Probing Negation Understanding in Vision-Language Models With Object Attention

Read the original on arXiv AI →

The paper investigates how vision‑language models (VLMs) handle negation by comparing their attention patterns to human gaze when processing affirmative versus negated statements. Using synthetic and realistic datasets, the authors find that most VLMs do not mimic human-like attention and rely on mechanisms beyond attention, with PaliGemma2‑3B shifting focus to alternatives under negation and LLaVA‑1.5‑7B showing a left‑side spatial bias. The study highlights a performance gap between affirmative and negated queries across downstream tasks, underscoring the need for better negation processing in VLMs.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

Related stories

arXiv AI
Sep 2

Visual Attention Faithfulness in Vision-Language Models is Heterogeneous

The study investigates whether attention weights in Vision‑Language Models (VLMs) accurately reflect model reasoning for visual inputs. Using causal perturbation analysis, it identifies three distinct processing modes—Faithful‑Sufficient, Faithful‑Distributed, and Non‑Focal—indicating heterogeneous visual attention faithfulness. The research also shows that human‑annotated ground‑truth regions align with model attention in only about 60% of cases, highlighting a systematic divergence between model visual reliance and human intuition across VQA, document, and chart tasks.

By Xurui Song, Weishi Wang, Zhongqi Yue, Kuluhan Binici, Tao Bai, Hongxin Shao, Daniel Dahlmeier, Jun Luo
llmsnlpmultimodal
More like this →
arXiv Computer Vision
3d ago

Perception or Reasoning? Diagnosing and Tackling Reinforcement Learning Bottlenecks in VLMs

The paper introduces Perception and Reasoning Attribution (PARA), a method that quantifies how much of a vision‑language model’s errors can be fixed by better perception versus stronger reasoning. PARA reveals that reinforcement learning (RL) benefits more from improving reasoning than perception, and motivates a new group‑relative attention distillation objective called GRAD. GRAD directly guides weaker rollouts toward the visual attention patterns of the best rollouts, consistently boosting performance across multiple model scales and benchmarks.

By Zeyuan Yang, Zhuohao Chen, Jiaben Chen, Yao Zhang, Maohao Shen, Chuang Gan
llmsreinforcement-learningefficiencymultimodalbenchmarks
More like this →