arXiv:2604. 14888v3 Announce Type: replace-cross Abstract: Recent advances in vision language models (VLMs) offer reasoning capabilities, yet how these unfold and integrate visual and textual information remains unclear.
By Danae S\'anchez Villegas, Samuel Lewis-Lim, Nikolaos Aletras, Desmond Elliott
The paper "Seeing is Believing: Benchmarking and Mitigating the Interference of Language Priors on Visual Perception in MLLMs" identifies that language priors can override visual evidence in multimodal large language models, leading to inflated accuracy metrics. It introduces ViSiP-Bench, a benchmark using rare-but-real scenarios across six prior categories, a dynamic prior induction protocol, and three metrics that more accurately gauge prior resistance. Evaluation of 24 models shows pervasive prior adherence, little improvement with scale, and a tendency for larger models to either strongly override or firmly adhere to priors. The authors propose MAPS, a fine‑tuning approach that softens excessive prior reliance by targeting each model’s strongest priors with factually grounded knowledge.
whyItMatters":"The study highlights that current accuracy‑based evaluations overestimate models’ visual perception due to language priors, and offers a more nuanced benchmark and mitigation strategy to improve multimodal model reliability."
By Li Ma, Yan Zhuang, Jiawen Deng, Fuji Ren
arXiv:2608.22232v1 Announce Type: new
Abstract: Real-world situation appearances can deviate from their underlying physical states, challenging the reliability of multimodal large language models (ML...
By Zhiming Yang, Zhuoxi Xiong, Donglin Zhou, Wenjun Wei, Shiyao Cui, Jinqiao Shi
The paper introduces Humanity's Sixth Sense (HSS), a benchmark designed to evaluate intuitive visual reasoning in multimodal large language models. HSS covers a wide range of image and video inputs, organizes them under a structured taxonomy, and pairs each with human‑written prompts that probe implicit temporal, spatial, social, and abstract cues that humans can infer at a glance. Current state‑of‑the‑art models, including GPT‑6‑astra, achieve far lower accuracy than humans (53.6 % vs. 93.1 %), highlighting a significant gap in intuitive visual understanding.
By Xingang Guo, Jing Gu, Brian Jang, Renxiong Wang, Utkarsh Tyagi, Daniel Quigley, Steven Li, David Yan, Daniel Yue Zhang, Darvin Yi, Forrest Huang, HiJae Kim, Tianyi Zhang, Jared Lichtarge, Jihua Huang, Le Xue, Manan Tomar, Qiuyi Richard Zhang, Ruofei Yu, Seth Neel, Yaning Hu, Marcella Valentine, Xinzhe Jiang, Daniel Evans, Chenguang Wang, Dustin Tran, Tong Zhao, Yinfei Yang, Yunzhong He
arXiv:2607. 26769v1 Announce Type: cross Abstract: Multimodal large language models increasingly use sketches, annotations, tools, and intermediate images during reasoning, but it remains unclear whether they truly rely on these visual states.
By Siyu Yan, Zhuoran Yan, Haiying Xu, Panhao Zhou, Jingyu Chen, Chenhao Ji, Shuo Cao, Yongheng Zhang, Haoze Liu, Siyu Zhang, Xiwen Gu, Yihao Liu, Alex Jinpeng Wang
The paper shows that controlled visual in‑context learning can help vision‑language models (VLMs) overcome errors caused by training priors. By providing demonstrations, the models focus more on visual evidence, yielding significant accuracy gains on counterfactual examples while leaving prior‑aligned performance largely unchanged. This effect is observed across five VLM families and five vision‑centric tasks, indicating that many VLMs possess latent visual grounding abilities that standard tests may miss.
By Kun Wang, Xindi Wu, Sanghyuk Chun, Olga Russakovsky, Esin Tureci