VKnowU is a benchmark that tests multimodal large language models (MLLMs) on their grasp of visual knowledge—intuitive, human-like understanding of physical and social principles in videos. The benchmark contains 1,680 questions across 1,249 videos, covering eight core types of visual knowledge, and shows that current state‑of‑the‑art MLLMs still lag behind human performance, especially on world‑centric tasks. To address this gap, the authors release VKnowQA and VideoKnow+, a baseline model that incorporates visual knowledge via a See‑Think‑Answer framework and reinforcement learning, improving performance on VKnowU and related datasets.
By Tianxiang Jiang, Sheng Xia, Yicheng Xu, Linquan Wu, Xiangyu Zeng, Limin Wang, Yu Qiao, Yi Wang
The paper shows that controlled visual in‑context learning can help vision‑language models (VLMs) overcome errors caused by training priors. By providing demonstrations, the models focus more on visual evidence, yielding significant accuracy gains on counterfactual examples while leaving prior‑aligned performance largely unchanged. This effect is observed across five VLM families and five vision‑centric tasks, indicating that many VLMs possess latent visual grounding abilities that standard tests may miss.
By Kun Wang, Xindi Wu, Sanghyuk Chun, Olga Russakovsky, Esin Tureci
The paper "Seeing is Believing: Benchmarking and Mitigating the Interference of Language Priors on Visual Perception in MLLMs" identifies that language priors can override visual evidence in multimodal large language models, leading to inflated accuracy metrics. It introduces ViSiP-Bench, a benchmark using rare-but-real scenarios across six prior categories, a dynamic prior induction protocol, and three metrics that more accurately gauge prior resistance. Evaluation of 24 models shows pervasive prior adherence, little improvement with scale, and a tendency for larger models to either strongly override or firmly adhere to priors. The authors propose MAPS, a fine‑tuning approach that softens excessive prior reliance by targeting each model’s strongest priors with factually grounded knowledge.
whyItMatters":"The study highlights that current accuracy‑based evaluations overestimate models’ visual perception due to language priors, and offers a more nuanced benchmark and mitigation strategy to improve multimodal model reliability."
By Li Ma, Yan Zhuang, Jiawen Deng, Fuji Ren
arXiv:2605.05593v2 Announce Type: replace
Abstract: Despite the remarkable success of Multimodal Large Language Models (MLLMs) across diverse tasks, the internal mechanisms governing how they encode...
By Zehao Deng, Tianjie Ju, Zheng Wu, Liangbo He, Jun Lan, Huijia Zhu, Weiqiang Wang, Zhuosheng Zhang
arXiv:2606. 29416v1 Announce Type: cross Abstract: Can a vision model truly see an object, or does it only fit surface-level visual cues?
By Xingyu Peng, Junran Wu, Yue Hou, Zhongliang Qiao, Jiaheng Liu, Shangzhe Li, Jichang Zhao, Wenjun Wu, Xianglong Liu, Yongxin Tong, Li Dong, Ke Xu
arXiv:2603. 06054v2 Announce Type: replace-cross Abstract: The use of Vision-Language Models (VLMs) in automated driving applications is becoming increasingly common, with the aim of leveraging their reasoning and generalisation capabilities to handle long-tail scenarios.
By Nikos Theodoridis, Reenu Mohandas, Ganesh Sistu, Anthony Scanlan, Ciar\'an Eising, Tim Brophy