arXiv AI
Beyond What Meets the Eye: Unveiling Situational Illusions for Multimodal Large Language Models
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv AI
5d ago
Seeing vs. Believing: Evaluating the Language Bias of Open-Source MLLMs in Counter-Intuitive Scenes
The paper introduces CAIT, a benchmark of 400 synthetic scenes featuring counter‑intuitive actions that challenge multimodal large language models (MLLMs). Human participants and proprietary models like Claude and Gemini perform well, but standard open‑source instruction‑tuned MLLMs fail, largely due to a strong language prior that overrides contradictory visual evidence. The study shows that Chain‑of‑Thought reasoning can help but introduces new issues, while targeted fine‑tuning and structured prompting can reduce reliance on language priors and improve visual grounding.
arXiv AI
Aug 7
HoloCount: A Holistic Visual Counting Benchmark for MLLMs
arXiv:2607. 06420v1 Announce Type: cross Abstract: Visual counting is a fundamental pillar of multimodal intelligence, requiring a seamless integration of fine-grained grounding and spatial reasoning.
arXiv AI
3d ago
Can LVLMs Uncover the Truth Behind Visual Illusions? An Analysis of Perceptual and Reasoning Capabilities
The paper introduces Illusion-Reasoning, a benchmark that uses real-world visual illusion images paired with annotated question-answer pairs to evaluate Large Vision Language Models (LVLMs). It argues that current evaluations focus too narrowly on perception or specific domains, and that a joint assessment of perception and reasoning in open-world settings is lacking. Using this benchmark, the authors demonstrate that many LVLMs’ reasoning abilities are less advanced than previously claimed, offering new insights and directions for optimization.
arXiv AI
Jul 17
Contextualized Evaluation of Vision Language Models through Dynamic, Multi-turn Interactions
arXiv:2607. 14499v1 Announce Type: new Abstract: Multi-modal Large Language Models (MLLMs) have made substantial advances on benchmarks, yet their real-world effectiveness remains uncertain.
arXiv AI
Jul 31
See2Think: Do Multimodal Models Really Use Intermediate Visual States?
arXiv:2607. 26769v1 Announce Type: cross Abstract: Multimodal large language models increasingly use sketches, annotations, tools, and intermediate images during reasoning, but it remains unclear whether they truly rely on these visual states.
arXiv Machine Learning
Jul 1
CoMet: Context and Multiplicity Decomposition for Multimodal Uncertainty Estimation
arXiv:2606. 32012v1 Announce Type: new Abstract: Uncertainty estimation has been a long-standing challenge in AI models; it amounts to "knowing what you don't know," and metacognition is notoriously difficult even for humans (cf.