arXiv Computer Vision

CitySTAR: Structured and Topology-Aware Reasoning for Open-Vocabulary Urban 3D Grounding

CitySTAR introduces a training‑free framework that transforms billion‑scale urban point clouds into a query‑ready scene graph of open‑vocabulary 3D instances, using CodeLLM‑driven tools to supply multimodal evidence for node attributes and spatial relations. It models target‑context topology with paired hypergraphs and performs bidirectional topology verification for structural disambiguation, followed by a Reflective Cross‑modal Grounding module that integrates topology consistency and 2D visual evidence to decide over a metric‑aware 3D context graph. The authors also present CitySTAR‑3D, a benchmark that enhances semantic coverage, instance completeness, bounding‑box fidelity, and spatial‑relation complexity for city‑scale 3D grounding, and report extensive experiments showing consistent improvements in open‑world urban 3D grounding with strong interpretability and generalization.

arXiv Computer Vision

6d ago

3D-SceneQ: Empowering 3D LLM With Query-Guided Adaptive Pruning and Multi-Modal Feature Enhancement

3D-SceneQ is a new 3D large language model that tackles two key limitations of existing 3D scene understanding pipelines: the inclusion of irrelevant object tokens that add noise, and the exclusive use of point cloud data that lacks rich semantic cues. It introduces a Query‑Guided Adaptive Pruning (QGAP) module that selectively keeps task‑relevant objects based on user instructions, and a Multi‑modal Object‑level Feature Enhancement (MOFE) module that fuses 2D feature embeddings into the 3D representation to enrich semantic information. Experiments show that 3D‑SceneQ surpasses state‑of‑the‑art methods on several benchmarks, achieving significant gains in metrics such as CIDEr and EM@1.

By Yongqi Shan, Lu Zhang, Jiazuo Yu, Yunzhi Zhuge, Huchuan Lu

llms rag efficiency multimodal benchmarks safety

arXiv Computer Vision

Oct 21

HERMES++: Toward a Unified Driving World Model for 3D Scene Understanding and Generation

HERMES++ is a unified driving world model that simultaneously performs 3D scene understanding and future geometry prediction. It uses a BEV representation to merge multi‑view spatial data for LLM compatibility, introduces LLM‑enhanced world queries for knowledge transfer, and employs a Current‑to‑Future Link to condition future geometry on semantic context. A Joint Geometric Optimization strategy enforces structural integrity by combining explicit geometric constraints with implicit latent regularization, and extensive benchmark tests show that HERMES++ outperforms specialist methods in both point cloud prediction and scene understanding.

By Xin Zhou, Dingkang Liang, Xiwu Chen, Feiyang Tan, Dingyuan Zhang, Hengshuang Zhao, Xiang Bai

llms agents robotics benchmarks

arXiv Computer Vision

Sep 28

GraphWrit3R: End-to-End 3D Scene Graph Writing

arXiv:2609.31595v1 Announce Type: new Abstract: 3D scene graphs provide a structured representation of complex environments by encoding objects, their semantic attributes, and the spatial and functio...

By Luka Milivojevic, Nikola Popovic, Sayan Deb Sarkar, Sebastian Koch, Iro Armeni, Luc Van Gool, Danda Pani Paudel

llms benchmarks safety

arXiv Computer Vision

Oct 27

GeoLift: Grounding Video Spatial Reasoning in Explicit 3D Maps

arXiv:2610.33558v1 Announce Type: new Abstract: Video spatial reasoning requires multimodal large language models (MLLMs) to understand how objects are arranged and related across viewpoints. While t...

By Xinglin Li, TingTing Long, Jingzhi Zhou, Jiajing Chen, Jian Yang, Jin Xie

llms multimodal

arXiv AI

Sep 4

GraFT: A Training-Free Framework for Spatial Reasoning in Multimodal Large Language Models via 3D Scene Graphs

GraFT is a training‑free framework that enhances spatial reasoning in multimodal large language models by integrating a compact 3D scene graph (3DSG). It offers deterministic geometry via symbolic tools, allocentric layout through bird’s‑eye‑view rendering, and visual‑attribute grounding using egocentric frames. Experiments on ScanQA and VSI‑Bench show significant performance gains, with CIDEr increasing by 27% and improvements up to 65% over baseline models.

By Junqing Du, Fernando Ropero, Erkin Turkoz, Yanfeng Zhang, Lu Liu

llms fine-tuning multimodal

arXiv AI

Sep 16

SceneBench: A Hierarchical Benchmark for Vision-Language Understanding of 3D Scenes

arXiv:2609.16233v1 Announce Type: cross Abstract: Vision-language models excel at 2D image understanding but remain limited in 3D spatial reasoning. Progress is hindered by limitations in current ben...

By Anubhav Khanal, Prabigya Acharya, Roshni Poudel, Sujan Kapali, Bigyan Bhatta, Pramish Paudel, Francois Rameau, Danda Pani Paudel

llms multimodal benchmarks

arXiv AI

Jul 8

OpenGround: Planning-based Online Perception for Open-World 3D Visual Grounding

arXiv:2512. 23020v3 Announce Type: replace-cross Abstract: 3D visual grounding aims to locate objects based on natural language descriptions in 3D scenes.

By Wenyuan Huang, Zhenyu Zhang, Zhao Wang, Zhou Wei, Ting Huang, Fang Zhao, Jian Yang

llms benchmarks

arXiv AI

Oct 2

Task-Adaptive Grounded 3D-Programmers Using 2D VLMs

The paper introduces 3D-Prog, a framework that adapts powerful 2D vision‑language models (VLMs) for reliable 3D understanding, manipulation, and generation. It does so by employing Canonical Coordinate Framing (CCF) to anchor inputs and outputs in a shared Euclidean coordinate system, resolving axis ambiguity, metric scale, and reference issues, and Task‑Adaptive Feedback (TAF) to close the reasoning loop with dynamic, task‑specific feedback. Together, CCF and TAF enable 2D VLMs to perform open‑vocabulary 3D tasks without retraining, yielding consistent, interpretable, and high‑quality results across diverse 3D scenarios.

By Arman Raayatsanati, Sombit Dey, Anna-Maria Halacheva, Jan-Nico Zaech, Luc Van Gool, Danda Pani Paudel

llms robotics multimodal

arXiv AI

Jul 1

PruneGround: Plug-and-play Spatial Pruning for 3D Visual Grounding

arXiv:2606. 31148v1 Announce Type: cross Abstract: 3D Visual Grounding (3DVG) aims to localize target objects in 3D scenes given natural language descriptions.

By Duc Cao Dinh, Khai Le-Duc, Florent Draye, Chris Ngo, Terry Jingchen Zhang, Bernhard Sch\"olkopf, Zhijing Jin

llms efficiency multimodal benchmarks

arXiv AI

Jul 9

SpaR3D-MoE: Adaptive 3D Spatial Reasoning from Sparse Views Meets Geometry-Inductive Mixture-of-Experts

arXiv:2607. 06620v1 Announce Type: cross Abstract: Recent Multimodal Large Language Models (MLLMs) struggle to bridge the representational gap between 2D semantic understanding and 3D spatial geometry.

By Haida Feng, Hao Wei, Haolin Wang, Shiwei Li, Chade Li, Yihong Wu

llms multimodal benchmarks

Hugging Face Trending Papers

Sep 14

SceneBench: A Hierarchical Benchmark for Vision-Language Understanding of 3D Scenes

Vision-language models excel at 2D image understanding but remain limited in 3D spatial reasoning. Progress is hindered by limitations in current benchmarks. First, 3D datasets often rely on point clo...

llms multimodal benchmarks

arXiv Computer Vision

Oct 21

Scene-R1: Video-Grounded Large Language Models for 3D Scene Reasoning with Lightweight Annotations

arXiv:2506.17545v2 Announce Type: replace Abstract: Large language models are increasingly being applied to 3D scene understanding. However, existing 3D-aware LLMs largely operate as black boxes: the...

By Junjie Wang, Zhihao Yuan, Shuyi Jiang, Chun-Mei Feng, Yaolun Zhang, Shuguang Cui, Zhen Li, Na Zhao

llms computer-vision nlp

Related stories