The paper introduces “SpaceConflict”, a benchmark of 23,196 multimodal inputs designed to test whether large language models can not only report spatial facts but also use them in reasoning. Experiments show a gap between a model’s ability to recover an initial spatial state from visual evidence and its ability to apply that state to perform transformations, with the gap narrowing but not closing as model size increases. To address this, the authors propose Operational State Supervision (OSS), which supervises task‑relevant spatial states and their transformation trajectories, improving accuracy on judgments that require state organization and use.
arXiv:2606.01914v2 Announce Type: replace Abstract: Multimodal large language models (MLLMs) remain unreliable on spatial multiple-choice questions, and their failures are often attributed to poorly...
arXiv:2605.30557v2 Announce Type: replace-cross Abstract: Spatial reasoning benchmarks typically evaluate whether vision-language models can derive the correct answer from a visual observation. Yet i...
arXiv:2606. 09669v1 Announce Type: new Abstract: Spatial reasoning is a foundational capability for multimodal large language models (MLLMs) to perceive and operate within the physical world.
STRAND is a new benchmark that tests multimodal large language models’ ability to track objects, their states, and relationships over time in videos. It evaluates intermediate reasoning by breaking queries into sub‑questions and uses Faithful Accuracy to ensure all parts of an answer are correct. The authors also propose an object‑centric framework that builds structured trajectories and shows reduced hallucinations and better temporal consistency compared to existing models.
arXiv:2607. 26769v1 Announce Type: cross Abstract: Multimodal large language models increasingly use sketches, annotations, tools, and intermediate images during reasoning, but it remains unclear whether they truly rely on these visual states.
arXiv:2606. 03988v1 Announce Type: new Abstract: Vision language models (VLMs) excel at many tasks but still struggle with spatial reasoning when critical information is not directly observable.
arXiv:2506.09557v2 Announce Type: replace-cross Abstract: While Multimodal Large Language Models (MLLMs) demonstrate impressive performance in benign scenarios, their cognitive reliability deteriorat...
arXiv:2608. 10954v1 Announce Type: cross Abstract: While Multimodal Large Language Models (MLLMs) demonstrate impressive performance in benign scenarios, their cognitive reliability deteriorates significantly in complex scenes under adverse conditions.
StateSight is a new benchmark designed to isolate and evaluate the ability of vision‑language models to reconstruct latent spatial structure from a single image. It consists of three procedurally generated task families—cube‑net opposite‑face reasoning, occluded cube‑tower counting, and 4‑neighbor connected‑component counting—each with 300 deterministic prompts and exact‑match scoring. The benchmark also includes a companion dataset, StateSight‑Steps, with 900 image‑text examples and 3,600 intermediate visual states to aid analysis of reconstruction errors.
arXiv:2606. 19965v1 Announce Type: cross Abstract: Multimodal large language models (MLLMs) are increasingly expected to act on visual information, yet the same scene may require different actions under different task contexts.
Aphanta is an automated framework that diagnoses how well image editors can produce task‑aligned visual intermediates for multimodal large language models (MLLMs). It evaluates three reasoning conditions—direct, editor‑generated, and idealized intermediate—to distinguish visual potential from practical editor performance across 20 tasks and various editor–MLLM pairs. The study finds that image editing benefits certain tasks like visual cue injection and grounding, but is less reliable for symbol‑sensitive or structural tasks, and demonstrates measurable performance gains with a Qwen pipeline.
arXiv:2606. 17389v1 Announce Type: cross Abstract: Multimodal Foundation Models are increasingly used as reasoning agents, making reliability, knowing when a model may hallucinate, critical.