← Back to all news
arXiv Computation and Language By Jiacheng Ding, Yanliang Qi, Cong Guo, Peihua Mai

See, Think, Use: Diagnosing Perception and Reasoning Failures in VLMs

Read the original on arXiv Computation and Language →

The paper introduces the See, Think, Use protocol, which evaluates vision‑language models by tracing each answer back to perception, reasoning, or the use of perception. It employs two additional questions per item—a localization probe for perception and a text‑replacement probe for reasoning—without requiring model weights. The protocol measures how often answers rely on the model’s own perception across counterfactual image pairs, and demonstrates that allocating compute to the failing stage can improve performance at lower cost.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computation and Language.

Related stories

arXiv AI
Oct 21

CounterCount: Diagnosing Knowledge Bias in How Vision-Language Models Count Attributes

CounterCount is a diagnostic dataset that pairs factual and counterfactual images to test whether Vision‑Language Models (VLMs) rely on visual evidence or on world knowledge when counting attributes. Humans achieve near‑perfect accuracy on both types of images, whereas VLMs drop from 91.6% to 49.5% on counterfactuals, often defaulting to canonical counts. The study shows that VLMs exhibit a System‑1‑like bias, underutilize visual evidence, and that targeted input manipulations can mitigate this bias.

By Reem Alzahrani, Bushra Bin Hemid, Hassan Alshanqiti, Zaid Alyafeai, Paul Gavrikov, Bernard Ghanem, Abdelrahman Eldesokey
llmsmultimodalsafety
More like this →
arXiv Computation and Language
Oct 22

Asking What a Director Sees Can Change What Vision-Language Models Choose

The paper investigates how asking vision‑language models to report what a director sees can alter their subsequent action choices. In a Director Task, prompting the model to identify visible blocks before selecting a block increases the likelihood of choosing a hidden block by up to 88 percentage points across several GPT models and 17 points for Qwen3.8‑27B, despite accurate reports of visible blocks. The authors argue that the question shifts the model’s perspective from the director’s to its own, and they propose a baseline paired‑prompt design for future perspective‑taking benchmarks.

By Jiaxuan Guo, Xingyu Ren, Kejia Zhang, Jingxin Yang, Youran Sun, Haizhao Yang
llmsmultimodalbenchmarks
More like this →
arXiv Machine Learning
Sep 11

New Evidence, Same Choice: Testing Physical Experiment Selection in Vision Language Models

The paper introduces a new benchmark for vision‑language models that tests their ability to decide whether to answer a physics question immediately or to request additional experimental evidence. Each problem presents one measurement image and four possible physical worlds defined by two masses and two values of another property; the model must either stop and answer or choose the cheapest experiment that resolves the question. Across six open models and 144 parameter sets, the models almost always repeat the same action even when the optimal choice changes, and only a single model gets both decisions correct on 5.9% of cases.

By Sourajit Saha, Shubhashis Roy Dipta, Nobin Sarwar, Shaswati Saha, Yuxuan Jiang, Siyuan Li, Qiheng Wang
llmsbenchmarks
More like this →