Matched Outcomes, Divergent Gaze: How Foveated MLLMs Search Compared to Humans
arXiv:2608. 16514v1 Announce Type: cross Abstract: Human visual search is serial: the fovea must land on a candidate to confirm it, and those landings form a scanpath.
The study compares human and vision‑language model (VLM) responses to cross‑modal association tasks, using identical stimuli (a pseudo‑word and two images) and recording both choices and eye movements. While larger VLMs show some alignment with human choices, their attention patterns correlate poorly with human gaze, performing no better than a simple center‑bias baseline. Fine‑tuning VLMs on human choices improves choice alignment but not attention alignment, and training on human gaze improves attention correlation without affecting choice accuracy.
arXiv:2608. 16514v1 Announce Type: cross Abstract: Human visual search is serial: the fovea must land on a candidate to confirm it, and those landings form a scanpath.
arXiv:2609.05517v1 Announce Type: cross Abstract: Human observers prioritize visual information according to task goals. Most computational models of naturalistic viewing are gaze-trained for free vi...
arXiv:2308. 06035v4 Announce Type: replace Abstract: Humans routinely draw on visual context to predict upcoming words.
arXiv:2606. 14703v1 Announce Type: cross Abstract: How a vision-language model internally solves the task of describing an image is far from obvious.
arXiv:2608.07282v2 Announce Type: replace Abstract: A central question for computational psycholinguistics is to what extent neural language models can serve as cognitively adequate models of human l...
The paper investigates how vision‑language models (VLMs) handle negation by comparing their attention patterns to human gaze when processing affirmative versus negated statements. Using synthetic and realistic datasets, the authors find that most VLMs do not mimic human-like attention and rely on mechanisms beyond attention, with PaliGemma2‑3B shifting focus to alternatives under negation and LLaVA‑1.5‑7B showing a left‑side spatial bias. The study highlights a performance gap between affirmative and negated queries across downstream tasks, underscoring the need for better negation processing in VLMs.
The study investigates whether attention weights in Vision‑Language Models (VLMs) accurately reflect model reasoning for visual inputs. Using causal perturbation analysis, it identifies three distinct processing modes—Faithful‑Sufficient, Faithful‑Distributed, and Non‑Focal—indicating heterogeneous visual attention faithfulness. The research also shows that human‑annotated ground‑truth regions align with model attention in only about 60% of cases, highlighting a systematic divergence between model visual reliance and human intuition across VQA, document, and chart tasks.
The paper investigates whether layer-wise visual‑text similarity in multimodal large language models (MLLMs) truly reflects content‑level cross‑modal interaction. By injecting Gaussian noise into the visual stream of 13 MLLMs, the authors show that task accuracy drops sharply while traditional scalar alignment metrics (CKA, SVCCA, MIR, principal‑angle cosine) fail to distinguish corrupted from clean inputs, a phenomenon they term the alignment illusion. They propose the principal‑angle gap (PA gap) as a more reliable geometric diagnostic that correlates better with task performance and reveals when internal geometry diverges from accuracy.
The paper proposes a new reward modeling approach that uses a spatially graded distribution of visual attention centered on the current gaze location, rather than just fixation points. By incorporating this gaze distribution into reinforcement learning from human feedback, the authors achieve higher preference prediction accuracy compared to a baseline. The results also qualitatively align with known patterns of human attention during reading, indicating that the model captures cognitively meaningful aspects of attention allocation.
arXiv:2607. 04590v1 Announce Type: new Abstract: Pairwise human comparisons are a primary interface through which modern AI systems learn human preferences.
arXiv:2606. 31054v1 Announce Type: cross Abstract: Multimodal Large Language Models (MLLMs) are critically hampered by hallucination, generating content inconsistent with the provided image.
arXiv:2609.18011v1 Announce Type: new Abstract: In collaborative tasks with asymmetric information, participants coordinate their understanding through interaction. We ask whether gaze provides evide...