arXiv AI By Sumin Hong, Katsumi Ibaraki, Renee Shi, David Chiang, Toby Jia-Jun Li

Similar Choices, Different Attention: Cross-Modal Associations in Humans and Vision-Language Models

Read the original on arXiv AI →

The study compares human and vision‑language model (VLM) responses to cross‑modal association tasks, using identical stimuli (a pseudo‑word and two images) and recording both choices and eye movements. While larger VLMs show some alignment with human choices, their attention patterns correlate poorly with human gaze, performing no better than a simple center‑bias baseline. Fine‑tuning VLMs on human choices improves choice alignment but not attention alignment, and training on human gaze improves attention correlation without affecting choice accuracy.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

arXiv AI
Oct 16

Probing Negation Understanding in Vision-Language Models With Object Attention

The paper investigates how vision‑language models (VLMs) handle negation by comparing their attention patterns to human gaze when processing affirmative versus negated statements. Using synthetic and realistic datasets, the authors find that most VLMs do not mimic human-like attention and rely on mechanisms beyond attention, with PaliGemma2‑3B shifting focus to alternatives under negation and LLaVA‑1.5‑7B showing a left‑side spatial bias. The study highlights a performance gap between affirmative and negated queries across downstream tasks, underscoring the need for better negation processing in VLMs.

By Beth Pearson, Michael Wray, Martha Lewis