arXiv AI September 2, 2026 By Yi-Cheng Lai, Hen-Hsen Huang

Do Multimodal LLMs See Before They Read? Diagnosing Contextual Sycophancy

The paper investigates a failure mode in multimodal large language models called multimodal contextual sycophancy, where external text can override conflicting image evidence. A diagnostic set of 998 cases independently varies visual evidence, commonsense priors, and external text to probe when this failure occurs. Experiments across six models show that a System‑2 Visual Arbitration (S2VA) approach, which withholds text from the visual witness, significantly improves performance over direct witness reports, with the best information boundary varying by model and context source.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

arXiv AI

Jul 15

Evidence Recomposition and Predictive Context Residualization for Visual Attribution in Multimodal Large Language Models

arXiv:2509. 22415v3 Announce Type: replace-cross Abstract: Multimodal large language models (MLLMs) have achieved strong vision-language performance, yet their token-level visual evidence remains difficult to inspect.

By Jiawei Liang, Jianjie Huang, Ruoyu Chen, Xianghao Jiao, Siyuan Liang, Shiming Liu, Xiaochun Cao

llms multimodal

arXiv AI

Jun 17

Vision-language models for chest radiography do not always need the image

arXiv:2606. 17710v1 Announce Type: cross Abstract: Medical vision-language models report strong chest radiograph accuracy, and this is increasingly read as evidence that they use the image.

By Mahshad Lotfinia, Sebastian Ziegelmayer, Lisa Adams, Daniel Truhn, Andreas Maier, Soroosh Tayebi Arasteh

llms multimodal benchmarks

arXiv AI

Jun 19

ROSE: Benchmarking the Perception-to-Action Gap in Multimodal Models

arXiv:2606. 19965v1 Announce Type: cross Abstract: Multimodal large language models (MLLMs) are increasingly expected to act on visual information, yet the same scene may require different actions under different task contexts.

By Yihao Wang, Zijian He, Jie Ren, Keze Wang

llms multimodal benchmarks

arXiv Machine Learning

3d ago

HalluPrism: When Multimodal Uncertainty Should Diagnose, Not Decide

arXiv:2608.29193v1 Announce Type: new Abstract: Multimodal Large Language Models (MLLMs) can assign similar confidence to answers that fail for different reasons. We propose HalluPrism, a behavioral...

By Aman Prakash, Sourish Dasgupta, Tanmoy Chakraborty

llms multimodal benchmarks

arXiv Computation and Language

2d ago

Same Semantics, Different Outcome: On the Modality Robustness of Multimodal LLMs under Knowledge Conflict

arXiv:2609.00550v1 Announce Type: new Abstract: Multimodal large language models (MLLMs) are increasingly provided with contextual evidence in heterogeneous forms: as a text passage, as a rendered im...

By Jungyeon Lee, Yejin Yoon, Taeuk Kim

llms rag fine-tuning multimodal safety

arXiv Computer Vision

Aug 27

Seeing or Knowing? Visual Context Sensitivity in Multimodal Large Language Models

The paper investigates why multimodal large language models (MLLMs) struggle with vision‑centric tasks when visual evidence conflicts with pretrained language knowledge. Using image reconstruction and a new WhatIfVis benchmark, the authors show that MLLMs preserve coarse‑grained visual attributes but fail to consistently use them, and that supervised fine‑tuning and activation patching can improve controllability of visual context sensitivity. The study demonstrates that the main bottleneck lies in the models’ inability to reliably regulate their reliance on visual evidence rather than in visual perception itself.

By Jiaang Li, Chengzu Li, Zhaochong An, Yifei Yuan, Xi Liu, Serge Belongie, V\'esteinn Sn{\ae}bjarnarson

llms fine-tuning multimodal benchmarks

One email a morning, machine-written

Related stories

Evidence Recomposition and Predictive Context Residualization for Visual Attribution in Multimodal Large Language Models

Vision-language models for chest radiography do not always need the image

ROSE: Benchmarking the Perception-to-Action Gap in Multimodal Models

HalluPrism: When Multimodal Uncertainty Should Diagnose, Not Decide

Same Semantics, Different Outcome: On the Modality Robustness of Multimodal LLMs under Knowledge Conflict

Seeing or Knowing? Visual Context Sensitivity in Multimodal Large Language Models