← Back to all news
arXiv Computer Vision By Li Ma, Yan Zhuang, Jiawen Deng, Fuji Ren

Seeing is Believing: Benchmarking and Mitigating the Interference of Language Priors on Visual Perception in MLLMs

Read the original on arXiv Computer Vision →

The paper "Seeing is Believing: Benchmarking and Mitigating the Interference of Language Priors on Visual Perception in MLLMs" identifies that language priors can override visual evidence in multimodal large language models, leading to inflated accuracy metrics. It introduces ViSiP-Bench, a benchmark using rare-but-real scenarios across six prior categories, a dynamic prior induction protocol, and three metrics that more accurately gauge prior resistance. Evaluation of 24 models shows pervasive prior adherence, little improvement with scale, and a tendency for larger models to either strongly override or firmly adhere to priors. The authors propose MAPS, a fine‑tuning approach that softens excessive prior reliance by targeting each model’s strongest priors with factually grounded knowledge. whyItMatters":"The study highlights that current accuracy‑based evaluations overestimate models’ visual perception due to language priors, and offers a more nuanced benchmark and mitigation strategy to improve multimodal model reliability."

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computer Vision.

Related stories

arXiv Computer Vision
Oct 12

In-Context Learning Can Help Vision-Language Models Overcome Training Priors

The paper shows that controlled visual in‑context learning can help vision‑language models (VLMs) overcome errors caused by training priors. By providing demonstrations, the models focus more on visual evidence, yielding significant accuracy gains on counterfactual examples while leaving prior‑aligned performance largely unchanged. This effect is observed across five VLM families and five vision‑centric tasks, indicating that many VLMs possess latent visual grounding abilities that standard tests may miss.

By Kun Wang, Xindi Wu, Sanghyuk Chun, Olga Russakovsky, Esin Tureci
llmsmultimodal
More like this →
arXiv AI
Aug 26

Seeing vs. Believing: Evaluating the Language Bias of Open-Source MLLMs in Counter-Intuitive Scenes

The paper introduces CAIT, a benchmark of 400 synthetic scenes featuring counter‑intuitive actions that challenge multimodal large language models (MLLMs). Human participants and proprietary models like Claude and Gemini perform well, but standard open‑source instruction‑tuned MLLMs fail, largely due to a strong language prior that overrides contradictory visual evidence. The study shows that Chain‑of‑Thought reasoning can help but introduces new issues, while targeted fine‑tuning and structured prompting can reduce reliance on language priors and improve visual grounding.

By Chen Ling, Tongwei Zhang, Hanqian Li, Nai Ding
llmsfine-tuningmultimodalbenchmarkssafety
More like this →
arXiv Computer Vision
Aug 27

Seeing or Knowing? Visual Context Sensitivity in Multimodal Large Language Models

The paper investigates why multimodal large language models (MLLMs) struggle with vision‑centric tasks when visual evidence conflicts with pretrained language knowledge. Using image reconstruction and a new WhatIfVis benchmark, the authors show that MLLMs preserve coarse‑grained visual attributes but fail to consistently use them, and that supervised fine‑tuning and activation patching can improve controllability of visual context sensitivity. The study demonstrates that the main bottleneck lies in the models’ inability to reliably regulate their reliance on visual evidence rather than in visual perception itself.

By Jiaang Li, Chengzu Li, Zhaochong An, Yifei Yuan, Xi Liu, Serge Belongie, V\'esteinn Sn{\ae}bjarnarson
llmsfine-tuningmultimodalbenchmarks
More like this →