← Back to all news
arXiv AI

Same Answer, Different Representations: Hidden instability in VLMs

Related stories

arXiv Computation and Language
Oct 12

When VLMs Destroy What They Know: Spatial Representations Fall Below Lexical Priors

The paper introduces EICS, a metric for assessing whether Vision‑Language Models (VLMs) consistently encode concepts under controlled one‑concept interventions, and CASP‑VL, a benchmark of causally controlled spatial perturbations derived from Visual Genome scene graphs. Experiments across multiple VLM architectures reveal that spatial relations are encoded far less faithfully than lexical information, even when models perform well on compositional benchmarks. The study shows that relational information is retained at the predicate token level but disappears when sentences are aggregated into a global representation, highlighting a systematic failure invisible to output‑based evaluation.

By Shubh Garg, Chen Shani
llmsmultimodalbenchmarks
More like this →
arXiv Computation and Language
Sep 24

PRISM-VLM: A Multi-Axis Discriminative Benchmark for Compact Vision-Language Models

PRISM‑VLM is a new benchmark for compact vision‑language models that evaluates each item across seven axes—task quality, behavioral robustness, and capability bottlenecks—rather than collapsing performance into a single accuracy score. It aggregates these axes into a single PScore while also providing per‑axis profiles, revealing differences that single‑axis benchmarks miss, such as sycophancy. The benchmark draws on items from fifteen public datasets and will be released with its full pipeline, prompts, and annotations.

By Sanghee Park, Kee-Eung Kim
llmsmultimodalbenchmarks
More like this →
arXiv AI
Oct 5

Corrupted but Correct: Why Vision-Language Models Lie to Themselves Internally

The paper demonstrates that a targeted adversarial perturbation can reduce a vision‑language model’s training loss to near zero for a fixed target caption, yet the same model, when generating freely, still produces the correct description. This phenomenon, termed the train/inference gap, is traced to a single autoregressive step where the target token’s rank is fixed across all images, and further analysis shows that the language decoder, rather than the visual encoder, determines whether the corrupted signal is amplified or suppressed. The study uses a controlled two‑stage PGD attack on Qwen2.5‑VL‑7B‑Instruct and evaluates the effect on 200 held‑out COCO images, revealing that adversarial robustness in autoregressive VLMs largely depends on the language decoder’s prior. whyItMatters":"The findings suggest that defenses and faithfulness evaluations for deployed vision‑language models should focus on the language decoder rather than the visual encoder, as the former is the key determinant of robustness to adversarial perturbations."

By Arun Josephraj Arokiaraj, Zekun Wu, Adriano Koshiyama
llmsmultimodalsafety
More like this →
arXiv Computation and Language
Sep 2

Reliability Challenges in Diffusion Vision-Language Models

The paper presents the first systematic reliability evaluation of diffusion-based Large Vision‑Language Models (dLVLMs), comparing six diffusion models to autoregressive (AR) baselines across four dimensions. Key findings include a reversal of the yes‑bias seen in AR models for binary visual queries, competitive hallucination rates but lower linguistic quality, near‑zero accuracy for underrepresented racial groups with opposite‑polarity gender bias, and accuracy collapse in multiple‑choice tasks when the correct option is shorter than distractors due to a length prior emerging at the first denoising step. Additionally, tokens committed late in denoising with low confidence correlate with hallucinated content, indicating a unique mechanistic signal in diffusion generation.

By Md. Atabuzzaman, Chris Thomas
llmsdiffusionmultimodalbenchmarkssafety
More like this →