arXiv AI

GoDeep: Annotation-Free Open-Vocabulary 3D Scene Understanding via Language-Space Lifting

GoDeep is an annotation‑free method for open‑vocabulary 3D scene understanding that uses a vision‑language model solely as a translator to generate structured, entity‑level descriptions of each image. These descriptions are projected and aggregated in a language‑only embedding space, eliminating the need for a 3D training corpus or domain‑specific encoder. The approach achieves competitive performance on ScanNet++ and a cultural heritage benchmark, accurately localizes out‑of‑vocabulary objects, and offers explainable, point‑level predictions.

arXiv AI

Aug 13

CORE-3D: Context-aware Open-vocabulary Retrieval by Embeddings in 3D

arXiv:2509. 24528v4 Announce Type: replace-cross Abstract: Object retrieval from a scene has become a new trend of research due to its numerous applications.

By Mohamad Amin Mirzaei, Pantea Amoie, Ali Ekhterachian, Matin Mirzababaei, Babak Khalaj

llms rag computer-vision multimodal benchmarks

arXiv Computer Vision

Aug 24

AffordAny: Open-World 3D Affordance Grounding from Monocular RGB Images via Vision-Language-Guided Geometric Reasoning

arXiv:2608.20720v1 Announce Type: new Abstract: Open-world 3D affordance grounding requires localizing functional object parts in 3D given free-form language queries. Existing methods typically assum...

By Junqi Wu, Kaihua Tang, Xuanwen Chen, Hongzhi Li, Jianqiang Huang, Xian-Sheng Hua

llms multimodal benchmarks

arXiv AI

Jul 31

TraceCLIP: Recovering Local Semantics from Patch-to-CLS Contributions

arXiv:2607. 26107v1 Announce Type: cross Abstract: Dense vision-language understanding, including object localization, region recognition, and open-vocabulary semantic segmentation, requires associating language concepts with spatially grounded visual regions.

By Xinran Liu, Shouqian Shi, Yutong Chen, Ge Wang, Xin-Wei Yao, Sheng Zhong

rag computer-vision multimodal benchmarks safety

Hugging Face Trending Papers

Jul 30

MonoVoc: Decoupling Geometry and Semantics for Lightweight Monocular Open-Vocabulary 3D Gaussians

Open vocabulary 3D scene understanding is essential for next-generation interactive systems, empowering users to intuitively query and navigate reconstructed environments using natural language. However, current 3D Gaussian frameworks are often bottlenecked by restrictive multiview capture requirements, costly scene-specific optimization, and the massive memory overhead of storing dense language features.

rag computer-vision nlp benchmarks

arXiv AI

Aug 25

ExtrinSplat: Decoupling Geometry and Semantics for Open-Vocabulary Understanding in 3D Gaussian Splatting

ExtrinSplat is a new framework that separates geometry from semantics in 3D Gaussian Splatting scenes. It clusters Gaussians into overlapping 3D object groups and uses a Vision‑Language Model to generate lightweight textual hypotheses, creating an extrinsic index layer that handles complex polysemy. This approach reduces adaptation time from hours to minutes, cuts storage overhead by orders of magnitude, and outperforms existing embedding‑based methods on open‑vocabulary 3D object selection and semantic segmentation benchmarks.

By Jiayu Ding, Xinpeng Liu, Zhiyi Pan, Shiqiang Long, Ge Li

llms rag computer-vision multimodal benchmarks

arXiv AI

Jun 19

QueryGaussian: Scalable and Training-Free Open-Vocabulary 3D Instance Retrieval

arXiv:2606. 19733v1 Announce Type: cross Abstract: Efficiently retrieving specific 3D instances from large-scale scenes via natural language prompts remains a formidable challenge in multimedia analysis.

By Xiuyuan Zhu, Ke Lu, Zijie Yang, Chao Yue, Jian Xue, Dongming Zhang

rag computer-vision efficiency benchmarks

arXiv AI

Jul 8

OpenGround: Planning-based Online Perception for Open-World 3D Visual Grounding

arXiv:2512. 23020v3 Announce Type: replace-cross Abstract: 3D visual grounding aims to locate objects based on natural language descriptions in 3D scenes.

By Wenyuan Huang, Zhenyu Zhang, Zhao Wang, Zhou Wei, Ting Huang, Fang Zhao, Jian Yang

llms benchmarks

arXiv AI

Jul 1

PruneGround: Plug-and-play Spatial Pruning for 3D Visual Grounding

arXiv:2606. 31148v1 Announce Type: cross Abstract: 3D Visual Grounding (3DVG) aims to localize target objects in 3D scenes given natural language descriptions.

By Duc Cao Dinh, Khai Le-Duc, Florent Draye, Chris Ngo, Terry Jingchen Zhang, Bernhard Sch\"olkopf, Zhijing Jin

llms efficiency multimodal benchmarks

arXiv Computer Vision

Aug 24

Stream3Dv2: Geometric-Semantic Fusion Enhanced Streaming Zero-Shot 3D Scene Understanding

arXiv:2608.21136v1 Announce Type: new Abstract: Recently, open-vocabulary zero-shot 3D scene understanding using vision foundation models has emerged as a promising alternative to data-intensive supe...

By Jie Xu, Na Zhao

llms agents computer-vision robotics

arXiv AI

Aug 12

Temporally Grounded Compositional Camera Motion Understanding via Geometric Knowledge Distillation

arXiv:2608. 10932v1 Announce Type: cross Abstract: Understanding camera motion is fundamental to video perception, with applications in spatial intelligence and controllable video generation.

By Dazhao Du, Shiyan Du, Jian Liu, Yongjian Yu, Bohai Gu, Tao Han, Hualuo Liu, Eric Liu, Yujia Zhang, Xi Chen, Song Guo

llms diffusion efficiency multimodal benchmarks

arXiv AI

Sep 4

ENEAS: Embedding-guided Neural Ensemble for Adaptive Segmentation

ENEAS is a unified, text‑promptable method that simultaneously provides precise instance tracking and high‑quality segmentation, and enables open‑concept discovery of any instance named by a text query. It extends the SeC architecture with a text‑prompting adapter and temporal memory to maintain targets through disappearance and avoid drifting, while a semantic verification layer combines visual embedding matching with conditional VLM refinement to filter ontological errors. Designed for 3D reconstruction, ENEAS delivers robust semantic tracking and segmentation across videos, libraries, and unordered collections, distinguishing true instances from look‑alike doppelgangers.

By Javier del Pino (SperidLabs), Salvador Rodr\'iguez (SperidLabs), Alejandro Garabito (SperidLabs), Javier \'Alvarez (SperidLabs), Chema Garabito (SperidLabs)

llms rag computer-vision multimodal safety

arXiv AI

Jul 8

PVCap: Towards Accurate 3D Dense Captioning via PseudoCap and VoxelCapNet

arXiv:2607. 06097v1 Announce Type: cross Abstract: 3D dense captioning, an emerging vision-language task, aims to generate descriptive sentences for each object in the 3D scene.

By Xiaopei Wu, Chenshu Hou, Liang Peng, Dan Xu, Binbin Lin, Xiaoshui Huang, Yuenan Hou, Yu Li, Wenxiao Wang, Haifeng Liu, Deng Cai, Wanli Ouyang

multimodal benchmarks

Related stories