arXiv:2610. 34855v1 Announce Type: new Abstract: Visual counting benchmarks report whether vision-language models produce the correct number, but not whether errors arise from perception or reasoning.
Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.
arXiv:2603. 06828v2 Announce Type: replace-cross Abstract: We uncover a behavioral law of long-horizon vision-language models: models that maintain temporally grounded beliefs generalize better.
PRISM-BENCH is a new benchmark comprising 1,000 expert‑curated vision‑language questions across five under‑represented reasoning categories, each paired with an idealized reasoning trace validated by a second annotator. It introduces diagnostics beyond final‑answer accuracy: a perception probe that attributes about 85% of frontier‑model failures to misperception, and trace alignment that predicts answer correctness with a macro AUROC of 0.66. The benchmark also presents CORE, an efficiency‑aware metric that reorders 24.2% of model pairs relative to accuracy while remaining uncorrelated with response length, revealing that recent gains are driven by recognition rather than spatial grounding.
CounterCount is a diagnostic dataset that pairs factual and counterfactual images to test whether Vision‑Language Models (VLMs) rely on visual evidence or on world knowledge when counting attributes. Humans achieve near‑perfect accuracy on both types of images, whereas VLMs drop from 91.6% to 49.5% on counterfactuals, often defaulting to canonical counts. The study shows that VLMs exhibit a System‑1‑like bias, underutilize visual evidence, and that targeted input manipulations can mitigate this bias.
The paper introduces Perception and Reasoning Attribution (PARA), a method that quantifies how much of a vision‑language model’s errors can be fixed by better perception versus stronger reasoning. PARA reveals that reinforcement learning (RL) benefits more from improving reasoning than perception, and motivates a new group‑relative attention distillation objective called GRAD. GRAD directly guides weaker rollouts toward the visual attention patterns of the best rollouts, consistently boosting performance across multiple model scales and benchmarks.
arXiv:2609.22588v1 Announce Type: new Abstract: Vision-language models (VLMs) perform strongly on visual question answering benchmarks, yet often make decisions that contradict visual evidence they h...
StateSight is a new benchmark designed to isolate and evaluate the ability of vision‑language models to reconstruct latent spatial structure from a single image. It consists of three procedurally generated task families—cube‑net opposite‑face reasoning, occluded cube‑tower counting, and 4‑neighbor connected‑component counting—each with 300 deterministic prompts and exact‑match scoring. The benchmark also includes a companion dataset, StateSight‑Steps, with 900 image‑text examples and 3,600 intermediate visual states to aid analysis of reconstruction errors.