← Back to all news
arXiv AI By Zhengxiang Wang, Owen Rambow

When Seeing Is Not Enough: Benchmarking Interactive Visual Grounding in LVLMs

Read the original on arXiv AI →

The paper introduces a controlled evaluation framework for interactive visual grounding in large vision-language models (LVLMs), examining how varying amounts of initial target information and dialogue affect performance. Experiments across four visual contexts and interaction protocols show that current LVLMs lag behind human baselines, especially when no initial description is given and information must be gathered through questions. The study also finds that LVLMs are poorly calibrated, often overestimating confidence, and that interactive grounding remains a significant challenge requiring visual matching, information seeking, and synthesis.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

Related stories

arXiv Computation and Language
3d ago

MagiC: Evaluating Multimodal Cognition Toward Grounded Visual Reasoning

MagiC is a benchmark for grounded multimodal cognition that evaluates vision‑language models on answer accuracy, step‑by‑step reasoning quality, grounding fidelity, and self‑correction. It contains about 5,500 weakly supervised QA examples generated from strong model outputs and 900 human‑curated examples with fine‑grained annotations, including rationales and bounding box groundings. The benchmark tests 15 models ranging from 7B to 70B parameters, introduces new metrics such as MagiScore and StepSense, and includes diagnostic settings to probe robustness against adversarial visual cues and introspective error correction.

By Chengfei Wu, Ronald Seoh, Bingxuan Li, Liqiang Zhang, Fengrong Han, Dan Goldwasser
llmsnlpmultimodalbenchmarkssafety
More like this →
arXiv AI
3d ago

Uncovering Grounding IDs: How External Cues Shape Multi-Modal Binding

The paper introduces Grounding IDs, latent identifiers that emerge when external visual cues—such as symbols or grid lines—structure both visual and textual modalities in large vision-language models. These IDs improve the alignment between visual components and their textual descriptions, enhancing cross‑modal grounding, reducing hallucinations, and boosting overall performance on multimodal tasks. The study demonstrates that partition‑inducing cues lead to more accurate connections and better attention between related components.

By Amirmohammad Izadi, Hosein Hasani, Fatemeh Askari, Mobin Bagherian, Sadegh Mohammadian, Mohammad Izadi, Mahdieh Soleymani Baghshah
llmsmultimodalsafety
More like this →