arXiv AI August 26, 2026 By Chen Ling, Tongwei Zhang, Hanqian Li, Nai Ding

Seeing vs. Believing: Evaluating the Language Bias of Open-Source MLLMs in Counter-Intuitive Scenes

The paper introduces CAIT, a benchmark of 400 synthetic scenes featuring counter‑intuitive actions that challenge multimodal large language models (MLLMs). Human participants and proprietary models like Claude and Gemini perform well, but standard open‑source instruction‑tuned MLLMs fail, largely due to a strong language prior that overrides contradictory visual evidence. The study shows that Chain‑of‑Thought reasoning can help but introduces new issues, while targeted fine‑tuning and structured prompting can reduce reliance on language priors and improve visual grounding.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

arXiv AI

Aug 6

Reasoning Dynamics and the Limits of Monitoring Modality Reliance in Vision-Language Models

arXiv:2604. 14888v3 Announce Type: replace-cross Abstract: Recent advances in vision language models (VLMs) offer reasoning capabilities, yet how these unfold and integrate visual and textual information remains unclear.

By Danae S\'anchez Villegas, Samuel Lewis-Lim, Nikolaos Aletras, Desmond Elliott

llms multimodal safety

arXiv AI

6d ago

Beyond What Meets the Eye: Unveiling Situational Illusions for Multimodal Large Language Models

arXiv:2608.22232v1 Announce Type: new Abstract: Real-world situation appearances can deviate from their underlying physical states, challenging the reliability of multimodal large language models (ML...

By Zhiming Yang, Zhuoxi Xiong, Donglin Zhou, Wenjun Wei, Shiyao Cui, Jinqiao Shi

llms fine-tuning multimodal benchmarks

arXiv AI

Jul 31

See2Think: Do Multimodal Models Really Use Intermediate Visual States?

arXiv:2607. 26769v1 Announce Type: cross Abstract: Multimodal large language models increasingly use sketches, annotations, tools, and intermediate images during reasoning, but it remains unclear whether they truly rely on these visual states.

By Siyu Yan, Zhuoran Yan, Haiying Xu, Panhao Zhou, Jingyu Chen, Chenhao Ji, Shuo Cao, Yongheng Zhang, Haoze Liu, Siyu Zhang, Xiwen Gu, Yihao Liu, Alex Jinpeng Wang

llms multimodal benchmarks

arXiv Machine Learning

Jun 8

Diagnosing Visual Ignorance in Vision-Language Models

arXiv:2606. 06890v1 Announce Type: cross Abstract: Vision-Language Models (VLMs) frequently rely on language priors, producing confident answers that are weakly grounded in visual evidence.

By Runyu Zhou, Qi Zhang, Qixun Wang, Yisen Wang

llms multimodal benchmarks

arXiv AI

Jul 17

Contextualized Evaluation of Vision Language Models through Dynamic, Multi-turn Interactions

arXiv:2607. 14499v1 Announce Type: new Abstract: Multi-modal Large Language Models (MLLMs) have made substantial advances on benchmarks, yet their real-world effectiveness remains uncertain.

By Yijiang Li, Huiqi Zou, Bingyang Wang, Ziang Xiao

llms multimodal benchmarks safety

arXiv AI

Aug 3

Step-Level Visual Grounding Faithfulness Predicts Out-of-Distribution Generalization in Long-Horizon Vision-Language Models

arXiv:2603. 06828v2 Announce Type: replace-cross Abstract: We uncover a behavioral law of long-horizon vision-language models: models that maintain temporally grounded beliefs generalize better.

By Md Ashikur Rahman, Md Arifur Rahman, Niamul Hassan Samin, Abdullah Ibne Hanif Arean, Juena Ahmed Noshin

llms multimodal benchmarks

One email a morning, machine-written

Related stories

Reasoning Dynamics and the Limits of Monitoring Modality Reliance in Vision-Language Models

Beyond What Meets the Eye: Unveiling Situational Illusions for Multimodal Large Language Models

See2Think: Do Multimodal Models Really Use Intermediate Visual States?

Diagnosing Visual Ignorance in Vision-Language Models

Contextualized Evaluation of Vision Language Models through Dynamic, Multi-turn Interactions

Step-Level Visual Grounding Faithfulness Predicts Out-of-Distribution Generalization in Long-Horizon Vision-Language Models