arXiv AI
Jun 19

SPOT-E: Test-Time Entropy Shaping with Visual Spotlights for Frozen VLMs

arXiv:2606. 20244v1 Announce Type: cross Abstract: Vision-language models (VLMs) often underperform on evidence intensive tasks because decisive visual evidence are small, localized, and easy to overlook, leading to failures in evidence readout even when high-level reasoning is intact.

By Bo Yin, Xiaobin Hu, Chengming Xu, Ruolin Shen, Mo Yang, Jiangning Zhang, Peng-Tao Jiang, Cheng Tan, Shuicheng YAN
arXiv Computer Vision
Sep 22

Look Where It Counts: A Free, Label-Free Visual Evidence Signal for Fine-Grained Vision-Language Reasoning

The paper introduces a free, label‑free visual evidence signal that improves fine‑grained vision‑language reasoning. By selecting image crops that maximize the model’s answer distribution peak, the method locates answer‑bearing regions without training or annotations, boosting accuracy from 70 % to 85 %. The evidence gap also complements model confidence, enabling better correctness prediction and error flagging.

By Santi Ram Tiwari, Nihal Naik, Devbrat Pandey, Nishant Sinha
arXiv AI
Oct 15

Visual Dependence Does Not Imply Answerability: A Systematic Analysis of Answerability Signals in Frozen InstructBLIP

The study evaluates whether signals from a frozen vision‑language model, InstructBLIP, can reliably indicate answerability in visual question answering. Using a 10K subset of UNK‑VQA, the authors test predictive uncertainty, cross‑modal alignment, visual grounding, visual dependence, internal representations, and learned probes, finding that all signals yield weak discrimination (AUROC 0.50‑0.54). Even interventions that remove question‑relevant visual evidence or increase probe capacity do not substantially improve answerability detection, underscoring that visual dependence and related metrics are not robust proxies for answerability.

By Minjun Kim, Hyunmin Song, Changwon Lim