← Back to all news
arXiv Computation and Language By Marcell Fekete, Johannes Bjerva, Tam\'as K\'aldi

When Discourse Pressures Conflict: Information Structure in Vision-Language Model Outputs

Read the original on arXiv Computation and Language →

The paper investigates how vision‑language models (VLMs) express visual content in discourse‑appropriate forms, focusing on information structure (IS) in Hungarian where Topic and Focus occupy distinct syntactic positions. By comparing six VLMs to human participants in visually grounded question answering, the authors find that VLMs produce IS‑relevant constructions but over‑regularise, collapsing onto narrow response templates, whereas humans use variable strategies influenced by discourse status, grammatical role, and definiteness. The study argues that VLM evaluation should extend beyond content accuracy to consider how content is packaged for discourse.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computation and Language.

Related stories

arXiv AI
Aug 26

When Seeing Is Not Enough: Benchmarking Interactive Visual Grounding in LVLMs

The paper introduces a controlled evaluation framework for interactive visual grounding in large vision-language models (LVLMs), examining how varying amounts of initial target information and dialogue affect performance. Experiments across four visual contexts and interaction protocols show that current LVLMs lag behind human baselines, especially when no initial description is given and information must be gathered through questions. The study also finds that LVLMs are poorly calibrated, often overestimating confidence, and that interactive grounding remains a significant challenge requiring visual matching, information seeking, and synthesis.

By Zhengxiang Wang, Owen Rambow
llmsmultimodalbenchmarks
More like this →
arXiv Computation and Language
Oct 5

Evaluating VQA in Vision Language Models using Cooperative Principles

The paper evaluates Vision Language Models (VLMs) on Visual Question Answering tasks where questions violate Grice's maxims. By generating question modifiers that add non-essential, ambiguous, or false information, the authors show that VLMs such as ChatGPT, Claude, Gemini, and Llava exhibit reduced performance. They also compare human pragmatic reasoning to VLM reasoning, noting differences in how each handles human‑induced versus AI‑generated violations, and find that humans spend less time resolving VLM‑induced violations while VLMs are less accurate in those cases.

By Monika Shah, Sudarshan Balaji, Somdeb Sarkhel, Sanorita Dey, Deepak Venugopal
llmsnlpmultimodal
More like this →
arXiv AI
Sep 2

Visual Attention Faithfulness in Vision-Language Models is Heterogeneous

The study investigates whether attention weights in Vision‑Language Models (VLMs) accurately reflect model reasoning for visual inputs. Using causal perturbation analysis, it identifies three distinct processing modes—Faithful‑Sufficient, Faithful‑Distributed, and Non‑Focal—indicating heterogeneous visual attention faithfulness. The research also shows that human‑annotated ground‑truth regions align with model attention in only about 60% of cases, highlighting a systematic divergence between model visual reliance and human intuition across VQA, document, and chart tasks.

By Xurui Song, Weishi Wang, Zhongqi Yue, Kuluhan Binici, Tao Bai, Hongxin Shao, Daniel Dahlmeier, Jun Luo
llmsnlpmultimodal
More like this →