arXiv AI
RealityBench: Simulated Environments to Evaluate a Vision Language Model's Ability to Document Human Physical Work
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv AI
Jul 2
MindEdit-Bench: Benchmarking Object-Level Counterfactual Spatial Reasoning in VLMs from In-the-Wild Photos
arXiv:2607. 00491v1 Announce Type: cross Abstract: Benchmarks for vision-language models (VLMs) mostly test observational spatial reasoning: models describe relations already visible in the input.
arXiv AI
6d ago
Where Do Vision Language Models Fail at 3D Reasoning?
The paper introduces BedBench, a benchmark of thirteen procedurally generated tasks designed to test vision‑language models’ ability to recover and consistently use exact 3D object properties. Across twelve open‑weight and four frontier models, the study finds a stark gap: humans achieve 93.3 % accuracy on per‑color counting, while the best model scores only 7.2 %. Even targeted fine‑tuning can raise a model’s accuracy to 68.6 % but leaves its answers about the most common colors inconsistent with its own per‑color counts, highlighting that recovering a property does not guarantee its consistent use.
arXiv Computer Vision
Sep 18
SceneTeract: Probing and Improving Agent-Aware Activity Reasoning in 3D Indoor Scenes
SceneTeract is a verification interface that separates semantic action understanding from physical feasibility in indoor 3D scenes. It decomposes activities into atomic actions and performs explicit geometric checks to determine executability, providing diagnostic traces for failures. The system reveals widespread functional and accessibility issues in synthetic scenes, shows that existing VLMs over‑predict action feasibility, and improves VLM performance through post‑training with verifier feedback, with benefits that generalize to real‑world scenes.
arXiv AI
Oct 1
WorldAuditBench: Interactive 3D World Auditing with Multimodal Agents
arXiv:2609.40325v1 Announce Type: new Abstract: As interactive 3D worlds are increasingly used to study intelligent behavior, it becomes important to develop efficient pipelines for identifying anoma...
arXiv Computer Vision
Sep 25
Seeing Is Not Measuring: Tool-Augmented Metric Spatial Reasoning for Vision-Language Models
The paper introduces a tool‑augmented framework that enhances a small Vision‑Language Model (Qwen3.5‑4B) with geometric tools—3D object detection, metric depth estimation, and deterministic solvers for distance, size, and bearing—to improve metric spatial reasoning. By moving metric computation from the model’s weights into explicit solvers, the approach achieves significant gains on ReVSI‑Bench tasks, notably increasing absolute distance accuracy from 0.46 to 0.74 MRA and relative direction accuracy from 25.9% to 73.4%. The modular design allows swapping in different detectors, enabling a clear separation between perception and reasoning errors, and the model can autonomously sequence the tools to match a scripted pipeline on most tasks.
arXiv Computer Vision
Sep 10
VANTAGE-Bench: Evaluating the Infrastructure AI Gap in Vision-Language Models
arXiv:2609.09396v1 Announce Type: new Abstract: As Vision-Language Models (VLMs) advance toward physical deployment, the focus has remained on action-oriented Embodied AI evaluated on subject-centric...