arXiv Computer Vision

OmniSpace: Efficient Geometry Awareness for Autonomous Vehicles MLLMs

OmniSpace is a plug‑and‑play framework that enhances multimodal large language models (MLLMs) with geometry awareness using only 2D observations. It introduces a Camera Pose Injector, a Multi‑view Epipolar Attention module, and a 3D Geometric Distillation objective to improve cross‑view correspondence and depth estimation. Experiments demonstrate that OmniSpace outperforms existing methods on planning, risk detection, language, and generalization benchmarks for autonomous vehicles.

arXiv Computer Vision

Oct 21

HERMES++: Toward a Unified Driving World Model for 3D Scene Understanding and Generation

HERMES++ is a unified driving world model that simultaneously performs 3D scene understanding and future geometry prediction. It uses a BEV representation to merge multi‑view spatial data for LLM compatibility, introduces LLM‑enhanced world queries for knowledge transfer, and employs a Current‑to‑Future Link to condition future geometry on semantic context. A Joint Geometric Optimization strategy enforces structural integrity by combining explicit geometric constraints with implicit latent regularization, and extensive benchmark tests show that HERMES++ outperforms specialist methods in both point cloud prediction and scene understanding.

By Xin Zhou, Dingkang Liang, Xiwu Chen, Feiyang Tan, Dingyuan Zhang, Hengshuang Zhao, Xiang Bai

llms agents robotics benchmarks

arXiv AI

Jul 24

3D-Aware VLMs with Implicit and Explicit Geometries

arXiv:2607. 21595v1 Announce Type: cross Abstract: Despite rapid progress, most existing vision-language models (VLMs) built from 2D visual inputs often struggle when handling various 3D tasks that require fine-grained spatial understanding and reasoning.

By Wenhao Li, Xueying Jiang, Quanhao Qian, Deli Zhao, Ran Xu, Shijian Lu, Gongjie Zhang

llms multimodal

arXiv Computation and Language

Sep 30

Imagine3D-LLM: Teaching MLLMs to Imagine 3D Scenes Before Answering

arXiv:2609.38177v1 Announce Type: cross Abstract: Reasoning about the 3D world from multi-view images remains a fundamental challenge for Multimodal Large Language Models (MLLMs). While modern MLLMs...

By Jaewoo Jung, Hyeonseo Yu, Honggyu An, Jisang Han, Mungyeom Kim, Minkyeong Jeon, Heeseong Shin, Wonjun Moon, Federico Tombari, Daniel Barath, Marc Pollefeys, Seungryong Kim, Sunghwan Hong

llms multimodal benchmarks

arXiv Computer Vision

6d ago

GeoLift: Grounding Video Spatial Reasoning in Explicit 3D Maps

arXiv:2610.33558v1 Announce Type: new Abstract: Video spatial reasoning requires multimodal large language models (MLLMs) to understand how objects are arranged and related across viewpoints. While t...

By Xinglin Li, TingTing Long, Jingzhi Zhou, Jiajing Chen, Jian Yang, Jin Xie

llms multimodal

arXiv Computer Vision

Oct 9

Seek-and-View Reasoning for Multi-View Spatial Understanding

The paper introduces Vantage, a training‑free, model‑agnostic framework that pairs a vision‑language model with a 3D foundation model to perform Seek‑and‑View reasoning for multi‑view spatial understanding. Vantage first analyzes a question to plan a relevant viewpoint, then augments reasoning with geometry‑grounded visual evidence from that viewpoint. Experiments across six VLMs on five benchmarks show consistent improvements without fine‑tuning, demonstrating that revealing spatial evidence through view‑grounded reasoning reduces reliance on language‑based cross‑view alignment.

By Qixiang Chen, Cheng Zhang, Fucai Ke, Chi-Wing Fu, Jianfei Cai, Jingwen Ye

llms fine-tuning multimodal benchmarks safety

arXiv AI

Jun 6

Learning Geometric Representations from Videos for Spatial Intelligent Multimodal Large Language Models

arXiv:2606. 05833v1 Announce Type: cross Abstract: Multimodal Large Language Models (MLLMs) excel at 2D semantic understanding but lack intrinsic 3D awareness, resulting in representations that fail to maintain geometric and spatial consistency across video frames.

By Haibo Wang, Lifu Huang

llms diffusion multimodal benchmarks

arXiv AI

Jun 4

DVGT: Driving Visual Geometry Transformer

arXiv:2512. 16919v2 Announce Type: replace-cross Abstract: Perceiving and reconstructing 3D scene geometry from visual inputs is crucial for autonomous driving.

By Sicheng Zuo, Zixun Xie, Wenzhao Zheng, Shaoqing Xu, Fang Li, Shengyin Jiang, Long Chen, Zhi-Xin Yang, Jiwen Lu

llms agents robotics safety

arXiv Computer Vision

Sep 1

Geometry-Grounded Unified 3D Perception for Autonomous Driving

arXiv:2608.13147v2 Announce Type: replace Abstract: Camera-based autonomous driving perception requires a shared representation that preserves metric 3D structure across synchronized multi-camera str...

By Longfei Xu, Xiaohui Wang, Zehao Huang, Han Li, Ya Yang, Naiyan Wang, Si Liu

agents computer-vision robotics benchmarks

arXiv AI

Jul 7

BEVLM: Distilling Semantic Knowledge from LLMs into Bird's-Eye View Representations

arXiv:2603. 06576v2 Announce Type: replace-cross Abstract: The integration of Large Language Models (LLMs) into autonomous driving has attracted growing interest for their strong reasoning and semantic understanding abilities, which are essential for handling complex decision-making and long-tail scenarios.

By Thomas Monninger, Shaoyuan Xie, Qi Alfred Chen, Sihao Ding

llms agents computer-vision robotics safety

arXiv Computer Vision

Sep 30

SphMind: Towards Robust, Training-Free VLM-based Spatial Reasoning with a 360 Camera

arXiv:2609.33462v2 Announce Type: replace Abstract: Omnidirectional or 360 cameras provide embodied AI agents with a holistic, wide field-of-view (FoV) view of their surroundings, motivating the use...

By Shriram Damodaran, Soumyaratna Debnath, Cheston Tan, Lin Wang

llms agents robotics multimodal benchmarks

arXiv Computation and Language

Oct 21

SpatialScout: Can VLM Agents Actively Explore and Map 3D Scenes?

SpatialScout is a benchmark that tests vision‑language model agents on active, object‑level 3D mapping. It includes 80 indoor scenes with 3,140 target objects, where agents must control a camera, gather RGB observations, and estimate object centers in a shared coordinate frame. Performance is measured by localization accuracy and target coverage using object‑scale overlap scores, revealing that current VLMs, including Claude Opus 5.5, struggle to build precise metric maps.

By Zhaoxi Zhang, Hao Liang, Linxuan Wu, Yiming Wu, Yifu Yuan, Kun Liang, Chenming Tang, Meiyi Qiang, Wentao Zhang, Weikang Li

llms agents nlp multimodal benchmarks

Hugging Face Trending Papers

Jun 4

Learning Geometric Representations from Videos for Spatial Intelligent Multimodal Large Language Models

Multimodal Large Language Models (MLLMs) excel at 2D semantic understanding but lack intrinsic 3D awareness, resulting in representations that fail to maintain geometric and spatial consistency across video frames. Given the scarcity of large-scale 3D data, we present GeoVR, a novel framework that learns geometric representations using purely 2D video sequences.

llms diffusion multimodal benchmarks

Related stories