arXiv AI

It's Not What the Image Shows: Irrelevant Context Destabilises VLM Judges Without Informing Them

arXiv AI
Oct 2

When the Judge Acts: Auditing VLM-Guided Image Selection on Culturally Situated Prompts

Vision‑language models (VLMs) are increasingly used as judges to select the best image from a set of generated options, yet their decisions are usually validated only by score agreement with human ratings rather than by the images they return. In this study, the authors audited VLM judges on 300 culturally situated prompts, comparing the returned image with unseen human ratings and with random choice, and repeated each decision with reordered candidates. They found that a 4‑B parameter judge barely outperforms random selection, shows significant position bias, and is highly sensitive to candidate ordering, while an 8‑B judge reduces position bias and outperforms a CLIP similarity baseline but still discards many good decisions when filtered.

By Huichan Seo
arXiv AI
Aug 14

How Do VLMs Behave When Blind or Misled? Behavioral Evaluation of VLMs on Scientific Figures

arXiv:2608. 13267v1 Announce Type: cross Abstract: Existing vision-language model (VLM) benchmarks emphasize perception and reasoning accuracy (how well VLMs describe and reason about what they see in an image), with limited attention to behavioral reliability under uncertainty (how they behave when visual evidence is missing or misleading).

By Paul Osemudiame Oamen, Owusu-Banahene Osei, Ananya Mukherjee, Christian Greisinger, Steffen Eger, Pius Onobhayedo, Wei Zhao
arXiv AI
Sep 28

The Uncontrolled Variable: Vision-Language Refusal Is Conditioned on the Image-Attachment Interface, and Not Robust to Irrelevant Image Properties

The paper demonstrates that vision‑language models’ refusal behavior is heavily influenced by whether an image is attached to a request, even when the image is blank or unreadable. Attaching such an image shifts refusal scores by large margins for borderline‑benign prompts while leaving genuinely neutral instructions largely unchanged. This effect varies with image properties, persists across checkpoints, and is not mitigated by explicit instructions to ignore the image.

By Haoyu Zhang, Yi Feng, Hanwen Liu, Shibo Zheng, Zhuoxi Wang, Yang Chen, Haowen Xu, Xiangchen Guan, Mohammad Zandsalimy, Shanu Sushmita
arXiv Computation and Language
Oct 19

Not Every Mention Is a Claim: Rethinking Object Hallucination Detection in Vision-Language Models

The paper argues that many vision‑language model (VLM) hallucination metrics mistakenly treat every object mentioned in a caption as a claim that the object is present. It introduces NAME‑Bench, a dataset of 25,497 non‑assertive mentions labeled as depictive, purpose, negation, or hedged, and shows that current detectors flag these as hallucinations at high rates. Adding non‑assertive mentions as negatives improves detection on LLaVA‑1.5 but requires much more data for newer VLMs, indicating that conflating mention with assertion is a persistent problem.

By Hsin-Ling Hsu, Neeraja Kirtane, Kuan-Hao Huang
arXiv AI
Oct 27

Conviction Without Evidence: Verdict Stability Is Independent of Visual Grounding in Medical Vision--Language Models

Vision–language models can correctly diagnose medical questions using only the clinical vignette, without the diagnostic image. In experiments on 60 Italian State Examination items, models defended diagnoses derived from a blank placeholder as firmly as those derived from real images, showing no increased fragility. While different adversarial pressures produced distinct dynamics—one leading to a single decisive flip and the other to continual changes—the instability under contentless doubt reflected item ambiguity rather than lack of evidence.

By Federico Felizzi, Francesco Andrea Causio, Bianca Destro Castaniti, Olivia Riccomi, Michele Ferramola, Vittorio De Vita, Alessandro Tosi, Antonio Cristiano, Alessia Longo, Lorenzo De Mori, Chiara Battipaglia, Melissa Sawaya, Luigi De Angelis, Marcello Di Pumpo, Alessandra Piscitelli, Pietro Eric Risuleo, Giulia Vojvodic, Mariapia Vassalli, Nicolo Scarsi, Manuel Del Medico
arXiv Computation and Language
Nov 4

Parsed Text Understates Vision-Language Models: From Below to Above Expert Agreement on Fine-Grained Facial Emotion

The paper introduces EmoNet‑Face‑HQ, a benchmark of 2,500 AI‑generated portraits each evaluated by four psychology experts across 40 emotion categories. It shows that most vision‑language models (VLMs) perform no better than random guessing unless fine‑tuned, but that changing the prompt to a simple yes/no per category can raise a VLM’s agreement with experts from chance to near expert levels. When evaluated using probability‑of‑yes scores on the five most‑agreed‑upon categories, eleven open‑weight VLMs achieve a quadratic‑weighted κ of 0.425, below the 0.468 two‑expert agreement, yet their average probability‑of‑yes score (0.553) matches the benchmark’s fine‑tuned model and outperforms it in three cases without any training.

By Tobias Hallmen, Fabian Deuser, Robin-Nico Kampa, Norbert Oswald, Elisabeth Andr\'e
arXiv AI
Sep 18

Cross-Modal Attention Acts as a Frequency Filter: Why Verbose Prompts Improve Robustness in Vision-Language Models

The paper shows that the wording of prompts in vision‑language models (VLMs) can either improve or worsen robustness to image corruption. Verbose prompts broaden the cross‑modal attention’s frequency filter, making the model less sensitive to corruptions, while semantically complex prompts narrow the filter and increase vulnerability. Experiments on Qwen3‑VL and LLaVA‑OneVision confirm that adding padding or verbose phrasing reduces answer drift by 70–81% on 8B models.

By Farooq Ahmad Wani, Maria Sofia Bucarelli, Mujtaba Hussain Mirza, Oleksandr Pryymak, Aryo Pradipta Gema, Iacopo Masi, Pasquale Minervini, Fabrizio Silvestri
arXiv Computer Vision
Oct 12

Auditing VLM Benchmarks from Existing Evaluation Runs

The paper introduces an audit layer for vision‑language model benchmarks that reuses per‑question artifacts from evaluation runs to assess the benchmark itself. Six detectors analyze saved outputs, and an additional text‑only rerun removes visual inputs to produce per‑question reports and aggregate statistics, including permutation‑null excess and interval estimates. Applying this method to the MMStar benchmark reveals that 20.2% of its questions show no accuracy gain from images, with a small but significant excess in mathematics and science categories, and identifies questions where all models chose the same non‑gold option.

By Denis Degtiarev, Anna Gromova, Sergey Katkov, Artem Mikhailin, Safina Mukhamedshina, Albert Nasybullin, Karina Shaikhutdinova, Artem Tuzov
arXiv Computer Vision
Nov 9

Did I See It or Think It? Probing Reality Monitoring in Vision-Language Models with Concept Injection

The paper investigates whether vision‑language models can distinguish between concepts that are internally injected and content actually present in an image. Using a new paradigm that independently varies the image and the injected concept, the authors test six VLMs and find that only Gemma‑3‑27B reliably identifies injected concepts, yet its source attribution is fragile. The study shows that injected concepts often appear in generated captions, highlighting the need to evaluate both content and source in multimodal self‑reports.

By Tina Khezresmaeilzadeh, Seyedarmin Azizi, Mehdi Kamal, Massoud Pedram