arXiv AI
Aug 10

Debias in Text, Believe Your Eyes: Text-Anchored Cross-Modal Transfer for Visual Counter-Commonsense Reasoning

arXiv:2608. 06938v1 Announce Type: cross Abstract: The visual reasoning ability of multimodal large language models (MLLMs) is crucial for downstream applications, particularly counter-commonsense reasoning, which requires models to reason beyond common assumptions.

By Chen Ling, Hanqian Li, Dongnan Liu, Keyu Qian, Jungang Li, Xinglong liu, Shiyi Wang, Xin Dong, Pengcheng Zhu, Wei Zhou, Linjian Mo, Nai Ding
arXiv Computer Vision
Nov 2

On the Faithfulness of Visual Thinking: Measurement and Enhancement

The paper investigates the faithfulness of visual information in multimodal chain-of-thought (MCoT) reasoning by large vision-language models (LVLMs). It shows that visual cues in MCoT are often inaccurate and largely ignored, as predictions change little when visual input is altered but significantly when textual input is changed. To improve visual faithfulness, the authors propose an annotation‑free Sufficient‑Component Cause Model (SCCM) that encourages the generation of minimal yet sufficient visual components, and demonstrate its effectiveness across fine‑grained perception and reasoning benchmarks.

By Zujing Liu, Junwen Pan, Qi She, Yuan Gao, Guisong Xia
arXiv AI
Sep 2

Visual Attention Faithfulness in Vision-Language Models is Heterogeneous

The study investigates whether attention weights in Vision‑Language Models (VLMs) accurately reflect model reasoning for visual inputs. Using causal perturbation analysis, it identifies three distinct processing modes—Faithful‑Sufficient, Faithful‑Distributed, and Non‑Focal—indicating heterogeneous visual attention faithfulness. The research also shows that human‑annotated ground‑truth regions align with model attention in only about 60% of cases, highlighting a systematic divergence between model visual reliance and human intuition across VQA, document, and chart tasks.

By Xurui Song, Weishi Wang, Zhongqi Yue, Kuluhan Binici, Tao Bai, Hongxin Shao, Daniel Dahlmeier, Jun Luo