arXiv Computer Vision By Tina Khezresmaeilzadeh, Seyedarmin Azizi, Mehdi Kamal, Massoud Pedram

Did I See It or Think It? Probing Reality Monitoring in Vision-Language Models with Concept Injection

Read the original on arXiv Computer Vision →

The paper investigates whether vision‑language models can distinguish between concepts that are internally injected and content actually present in an image. Using a new paradigm that independently varies the image and the injected concept, the authors test six VLMs and find that only Gemma‑3‑27B reliably identifies injected concepts, yet its source attribution is fragile. The study shows that injected concepts often appear in generated captions, highlighting the need to evaluate both content and source in multimodal self‑reports.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computer Vision.

arXiv Computation and Language
Nov 6

Vision-Default, Prior-Override: Causal Mechanisms of Perception-Knowledge Conflict in Vision-Language Models

The paper investigates how vision‑language models resolve conflicts between visual evidence and memorized knowledge. Using activation patching and ablation across residual streams, attention heads, and MLP sublayers, the authors find that visual grounding is the default behavior, while prior knowledge relies on a small, causally essential set of attention heads concentrated in the network’s second half. Ablating these heads flips knowledge‑grounded predictions to visually grounded ones in most cases, revealing an asymmetric causal structure and a sparse circuit of routing and writing heads that underlies perception‑knowledge conflict.

By Niclas Lietzow, Danielle Bitterman, Carsten Eickhoff, William Rudman, Michal Golovanevsky
arXiv AI
Oct 21

CounterCount: Diagnosing Knowledge Bias in How Vision-Language Models Count Attributes

CounterCount is a diagnostic dataset that pairs factual and counterfactual images to test whether Vision‑Language Models (VLMs) rely on visual evidence or on world knowledge when counting attributes. Humans achieve near‑perfect accuracy on both types of images, whereas VLMs drop from 91.6% to 49.5% on counterfactuals, often defaulting to canonical counts. The study shows that VLMs exhibit a System‑1‑like bias, underutilize visual evidence, and that targeted input manipulations can mitigate this bias.

By Reem Alzahrani, Bushra Bin Hemid, Hassan Alshanqiti, Zaid Alyafeai, Paul Gavrikov, Bernard Ghanem, Abdelrahman Eldesokey
arXiv AI
Sep 10

Counterfactual Tests for Measuring Chain-of-Thought Faithfulness in Visual Language Models

The paper introduces visual adaptations of counterfactual tests—vCT and vCCT—to evaluate whether chain-of-thought explanations in vision‑language models faithfully reflect the visual evidence driving predictions. Using these tests, the authors benchmark eight open‑source VLMs on two datasets and find that CoTs often fail to track visual evidence, sometimes omitting removed objects or mentioning them inconsistently. They also release two new datasets, Counter‑SNLI‑VE and Counter‑A‑OKVQA, consisting of image pairs that differ by a single object to facilitate further research.

By Bayar Menzat, Maximilian S\"uss, Ruizhi Wang, Benno Steinegger, Thomas Lukasiewicz, Oana-Maria Camburu