arXiv Computer Vision

What Do Hallucinations Reveal About Multimodal Reasoning? Diagnosing Visual Grounding Failures via Contrastive Decoding Probes

arXiv AI
Aug 20

ReWEIGH the Evidence: Calibrating Token-Level Ordinal Visual Evidence to Mitigate Hallucinations in Large Vision-Language Models

ReWEIGH the Evidence is a training‑free decoding technique that calibrates token‑level ordinal visual evidence to reduce hallucinations in large vision‑language models. It aggregates vocabulary ranks across visual positions, compares candidates to a token‑specific reference derived from unlabeled images, and applies a bounded penalty only when evidence falls below this reference. Experiments on four 7B backbones show up to a 21.3% reduction in hallucinated object mentions while largely preserving or improving descriptive and general performance, with minimal added latency.

By Jihae Jeong, Junha Choi, Hwanjo Yu
arXiv Computer Vision
Sep 3

Detecting Object Hallucinations in Large Vision-Language Models via Cross-Modal Attention Drifts and Mask-Based Verification

The paper introduces CADMP, a lightweight framework for detecting object hallucinations in large vision‑language models. CADMP measures cross‑modal attention drift between adjacent layers and verifies predictions by masking visually relevant regions, combining these signals to identify hallucinated outputs. Experiments on multiple benchmarks show that CADMP achieves competitive detection performance, and ablation studies confirm the complementary roles of attention drift and mask‑based verification.

By Xuanbing Wen, Boxu Chen, Le Yang, Jiakai Wang, Zhengyu Zhao, Chenhao Lin, Chao Shen