← Back to all news
arXiv AI By Gexin Huang, Haisheng Fu, Yanting Yang, Chen Zhao, Ruinan Jin, Jiaqi Liu, Xiaoxiao Li

VLMProbe: An Open-source Library for VLM Probing and Interpretation

Read the original on arXiv AI →

VLMProbe is an open‑source library that unifies lens readouts, attention attribution, causal interventions, and sparse autoencoder analysis for vision‑language models. It offers shared adapters, calibration tools, evaluation pipelines, and visualizations to enable systematic comparison across different model architectures. The framework demonstrates that representation readouts and spatial explanations vary with the analysis goal and observation point, and that local attribution can conflict with intervention outcomes, highlighting the need for multiple evidence sources in interpretation studies.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

Related stories

arXiv Computer Vision
Aug 27

Seeing or Knowing? Visual Context Sensitivity in Multimodal Large Language Models

The paper investigates why multimodal large language models (MLLMs) struggle with vision‑centric tasks when visual evidence conflicts with pretrained language knowledge. Using image reconstruction and a new WhatIfVis benchmark, the authors show that MLLMs preserve coarse‑grained visual attributes but fail to consistently use them, and that supervised fine‑tuning and activation patching can improve controllability of visual context sensitivity. The study demonstrates that the main bottleneck lies in the models’ inability to reliably regulate their reliance on visual evidence rather than in visual perception itself.

By Jiaang Li, Chengzu Li, Zhaochong An, Yifei Yuan, Xi Liu, Serge Belongie, V\'esteinn Sn{\ae}bjarnarson
llmsfine-tuningmultimodalbenchmarks
More like this →
arXiv Computer Vision
Oct 12

In-Context Learning Can Help Vision-Language Models Overcome Training Priors

The paper shows that controlled visual in‑context learning can help vision‑language models (VLMs) overcome errors caused by training priors. By providing demonstrations, the models focus more on visual evidence, yielding significant accuracy gains on counterfactual examples while leaving prior‑aligned performance largely unchanged. This effect is observed across five VLM families and five vision‑centric tasks, indicating that many VLMs possess latent visual grounding abilities that standard tests may miss.

By Kun Wang, Xindi Wu, Sanghyuk Chun, Olga Russakovsky, Esin Tureci
llmsmultimodal
More like this →
arXiv AI
Sep 17

EDCT-Bench: Uncovering Faithfulness Gaps in VLMs via Explanation-Driven Counterfactual Testing

EDCT-Bench is a benchmark that evaluates the faithfulness of Vision‑Language Models (VLMs) by using Explanation‑Driven Counterfactual Testing (EDCT). EDCT extracts visual concepts from a model’s natural language explanation, applies minimal verified edits to those concepts, and checks whether the model’s answer and explanation remain consistent with the edited image. The benchmark covers three domains—knowledge‑intensive VQA, safety‑critical driving, and 3D spatial reasoning—and reveals significant faithfulness gaps in current VLMs, while also showing that EDCT‑generated counterfactuals can improve training.

By Sihao Ding, Santosh Vasa, Aditi Ramadwar, Thomas Monninger
llmsnlpfine-tuningmultimodalbenchmarkssafety
More like this →