arXiv Computer Vision November 2, 2026 By Hao Yin, Tianchen Guo, Heming Du, Yan Ke, Yanbin Liu, Xin Yu

When Depth Lies: Benchmarking Vision-Language Models on Mirror-Induced RGB-D Ambiguity

Read the original on arXiv Computer Vision →

The paper introduces MIRA-Bench, a mirror‑centric RGB‑D dataset and benchmark designed to test vision‑language models (VLMs) in reflective scenes where depth information is misleading. It contains 1,011 RGB‑D images from 525 real‑world scenes, annotated with reflective surface masks and object labels for both direct and reflected views, and 14,140 QA pairs across eight sub‑tasks. Evaluation shows that state‑of‑the‑art VLMs consistently fail to distinguish real from reflected objects, misalign reflected instances with their real counterparts, and produce incorrect spatial inferences, with depth data even degrading performance. whyItMatters":"The study reveals that current VLMs cannot handle mirror‑induced depth ambiguity, highlighting a critical limitation in spatial reasoning for real‑world applications involving reflective surfaces."

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computer Vision.

arXiv AI

Jul 2

MindEdit-Bench: Benchmarking Object-Level Counterfactual Spatial Reasoning in VLMs from In-the-Wild Photos

arXiv:2607. 00491v1 Announce Type: cross Abstract: Benchmarks for vision-language models (VLMs) mostly test observational spatial reasoning: models describe relations already visible in the input.

By Leyuan Yu, Xiao Tang, Minghao Liu, Xinyuan Li, Xiaokai Bai, Sheng Zhou, Qunshu Lin, Weihao Xuan, Naoto Yokoya

llms multimodal benchmarks

arXiv AI

Jun 17

m2sv: A Scalable Benchmark for Map-to-Street-View Spatial Reasoning

arXiv:2601. 19099v2 Announce Type: replace-cross Abstract: Vision--language models (VLMs) achieve strong performance on many multimodal benchmarks but remain brittle on spatial reasoning tasks that require aligning abstract overhead representations with egocentric views.

By Yosub Shin, Michael Buriek, Igor Molybog

llms reinforcement-learning fine-tuning multimodal benchmarks safety

arXiv Computation and Language

Oct 9

SpaceCast-Bench: Evaluating Predictive Spatial Reasoning in Vision-Language Models

SpaceCast-Bench is a new benchmark that evaluates predictive spatial reasoning in vision‑language models, moving beyond simple spatial perception to tasks that require constructing scenes, anticipating interventions, and reasoning about unseen outcomes. It contains 3,862 questions from 182 real‑world scenes across 16 task types and three difficulty levels—static perception, local prediction, and global prediction—testing scene understanding, spatial state updating, and relational inference. Evaluation of 21 models shows a large performance gap, with the best model achieving only 58.0% versus 87.2% human accuracy, and fine‑tuning on the benchmark’s programmatically generated data can substantially improve model performance.

By Hongxing Li, Jinyue Su, Dingming Li, Wenqi Zhang, Weiming Lu, Jun Xiao, Yueting Zhuang, Yongliang Shen

llms fine-tuning multimodal benchmarks

arXiv AI

Sep 16

SceneBench: A Hierarchical Benchmark for Vision-Language Understanding of 3D Scenes

arXiv:2609.16233v1 Announce Type: cross Abstract: Vision-language models excel at 2D image understanding but remain limited in 3D spatial reasoning. Progress is hindered by limitations in current ben...

By Anubhav Khanal, Prabigya Acharya, Roshni Poudel, Sujan Kapali, Bigyan Bhatta, Pramish Paudel, Francois Rameau, Danda Pani Paudel

llms multimodal benchmarks

arXiv Machine Learning

Jun 19

Do Vision-Language Models Understand 3D Scenes or Just Catalogue Objects?

arXiv:2605. 20448v2 Announce Type: replace-cross Abstract: Vision-language models reliably name objects in a scene, but do they represent the 3D layout those objects inhabit?

By Animesh Maheshwari, Divyansh Sahu, Nishit Verma

llms robotics multimodal benchmarks

arXiv AI

Oct 7

Seeing Isn't Knowing: Do VLMs Know When Not to Answer Spatial Questions (and Why)?

arXiv:2605.30557v2 Announce Type: replace-cross Abstract: Spatial reasoning benchmarks typically evaluate whether vision-language models can derive the correct answer from a visual observation. Yet i...

By Yue Zhang, Zun Wang, Han Lin, Yonatan Bitton, Idan Szpektor, Mohit Bansal

llms fine-tuning multimodal benchmarks

One email a morning, machine-written

Related stories

MindEdit-Bench: Benchmarking Object-Level Counterfactual Spatial Reasoning in VLMs from In-the-Wild Photos

m2sv: A Scalable Benchmark for Map-to-Street-View Spatial Reasoning

SpaceCast-Bench: Evaluating Predictive Spatial Reasoning in Vision-Language Models

SceneBench: A Hierarchical Benchmark for Vision-Language Understanding of 3D Scenes

Do Vision-Language Models Understand 3D Scenes or Just Catalogue Objects?

Seeing Isn't Knowing: Do VLMs Know When Not to Answer Spatial Questions (and Why)?