The Flow has not summarised this story yet — read it at arXiv Computation and Language.
The study compares human and vision‑language model (VLM) responses to cross‑modal association tasks, using identical stimuli (a pseudo‑word and two images) and recording both choices and eye movements. While larger VLMs show some alignment with human choices, their attention patterns correlate poorly with human gaze, performing no better than a simple center‑bias baseline. Fine‑tuning VLMs on human choices improves choice alignment but not attention alignment, and training on human gaze improves attention correlation without affecting choice accuracy.
arXiv:2308. 06035v4 Announce Type: replace Abstract: Humans routinely draw on visual context to predict upcoming words.
Gaze Attention is a new mechanism for multimodal large language models that selectively focuses on relevant visual regions during each generation step, rather than attending to all visual tokens. By grouping tokens into spatial regions and using learnable context tokens to retain global information, it reduces attention computation and visual key‑value entries by up to 90%. Experiments on 13 image and 6 video benchmarks show that Gaze Attention matches or outperforms dense‑attention baselines while using fewer visual resources.
The paper explores using eye tracking to evaluate the visual realism of AI-generated videos, supplementing traditional explicit feedback. Two Visual Turing Test experiments with 90 participants produced 4,286 eye‑tracking trials across 438 videos, revealing that gaze patterns correlate with explicit judgments. The authors introduce GazeEval, a computational model that predicts perceived realism from eye‑tracking data with 71.3% accuracy, and they release the dataset for future research.
arXiv:2609.14207v1 Announce Type: new Abstract: We propose to finetune vision-language models to generate more pragmatically optimal referring expressions by transforming observations of incremental...
arXiv:2609.05517v1 Announce Type: cross Abstract: Human observers prioritize visual information according to task goals. Most computational models of naturalistic viewing are gaze-trained for free vi...