← Back to all news
arXiv AI By Zhengxiang Wang, Owen Rambow

When Seeing Is Not Enough: Benchmarking Interactive Visual Grounding in LVLMs

Read the original on arXiv AI →

The paper introduces a controlled evaluation framework for interactive visual grounding in large vision-language models (LVLMs), examining how varying amounts of initial target information and dialogue affect performance. Experiments across four visual contexts and interaction protocols show that current LVLMs lag behind human baselines, especially when no initial description is given and information must be gathered through questions. The study also finds that LVLMs are poorly calibrated, often overestimating confidence, and that interactive grounding remains a significant challenge requiring visual matching, information seeking, and synthesis.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

Related stories

arXiv AI
Jul 14

The Ebb and Flow of Multimodal Focus: Scheduling Visual Relay Windows for Grounded VLM Reasoning

arXiv:2607. 11436v1 Announce Type: new Abstract: Vision-language models increasingly succeed on multimodal reasoning benchmarks, yet their visual evidence often becomes unstable once it enters the language stack, weakening evidence-grounded reasoning.

By Wencheng Ye, Yi Bin, Yujuan Ding, Hongye Fang, Zheng Wang, Xing Xu, Jingkuan Song, Yun Zhang, Sirui Da, Heng Tao Shen
llmsmultimodalbenchmarks
More like this →
arXiv AI
2d ago

Towards Generalizable Visually Grounded Exploration of Household Devices

The paper introduces VGEBench, a new benchmark for evaluating Vision‑Language Models (VLMs) on generalizable, visually grounded exploration of household devices. Unlike existing datasets that rely on static images or annotated trajectories, VGEBench employs a logic‑driven state machine to simulate multi‑turn interaction loops, requiring agents to actively perceive, act, and refine their actions to achieve goals. Experiments show that current VLMs struggle to translate semantic knowledge into physical execution and to maintain long‑horizon state tracking.

By Linhao Zheng, Zeming Liu, Wangke Chen, Li Zeng, Wanxiang Che, Heyan Huang, Yuhang Guo
llmsagentsroboticsmultimodalbenchmarks
More like this →
Hugging Face Trending Papers
Jul 23

CRAG-MM-Diagnostics: Enabling Stage-Wise Analysis of Knowledge-Intensive VQA

Knowledge-Intensive Visual Question Answering (KI-VQA) benchmarks evaluate Vision-Language Models (VLMs) as multimodal knowledge assistants by requiring external information beyond a provided image to answer questions. KI-VQA involves multiple sub-problems -referring expression understanding, visual grounding, object recognition, knowledge retrieval, and reasoning-yet existing benchmarks typically report only end-task accuracy, obscuring where failures arise.

llmsragnlpmultimodalbenchmarks
More like this →