arXiv AI
Aug 13
CORE-3D: Context-aware Open-vocabulary Retrieval by Embeddings in 3D
arXiv:2509. 24528v4 Announce Type: replace-cross Abstract: Object retrieval from a scene has become a new trend of research due to its numerous applications.
GoDeep: Annotation-Free Open-Vocabulary 3D Scene Understanding via Language-Space Lifting
GoDeep is an annotation‑free method for open‑vocabulary 3D scene understanding that uses a vision‑language model solely as a translator to generate structured, entity‑level descriptions of each image. These descriptions are projected and aggregated in a language‑only embedding space, eliminating the need for a 3D training corpus or domain‑specific encoder. The approach achieves competitive performance on ScanNet++ and a cultural heritage benchmark, accurately localizes out‑of‑vocabulary objects, and offers explainable, point‑level predictions.
Related stories
arXiv Computer Vision
Aug 24
AffordAny: Open-World 3D Affordance Grounding from Monocular RGB Images via Vision-Language-Guided Geometric Reasoning
arXiv:2608.20720v1 Announce Type: new Abstract: Open-world 3D affordance grounding requires localizing functional object parts in 3D given free-form language queries. Existing methods typically assum...
arXiv AI
Jul 31
TraceCLIP: Recovering Local Semantics from Patch-to-CLS Contributions
arXiv:2607. 26107v1 Announce Type: cross Abstract: Dense vision-language understanding, including object localization, region recognition, and open-vocabulary semantic segmentation, requires associating language concepts with spatially grounded visual regions.
arXiv Computer Vision
Oct 2
LangMap: A Human-Verified Benchmark for Hierarchical Open-Vocabulary Goal Navigation
LangMap introduces a human‑verified benchmark for language‑conditioned goal navigation (LGN) that spans four hierarchical semantic levels—scene, room, region, and instance—within real‑world indoor 3D scans. The dataset, built on HM3D, contains 18K tasks with concise and detailed descriptions for 414 object categories, and its contrastive annotation protocol ensures high‑quality, discriminative region and instance labels. Evaluation shows that LangMap’s descriptions improve text‑to‑view matching accuracy by 23 points over GOAT‑Bench and achieve a 92.5% unique‑and‑correct match rate in an independent human audit, while a proposed RGB‑only baseline, PlaNaVid, attains top‑tier success rates without depth or 3D scene representations.
arXiv Computer Vision
6d ago
Query-Time Calibration of CLIP Features for Open-Vocabulary 3D Segmentation
The paper introduces a query‑time calibration technique for CLIP features lifted into 3D Gaussian reconstructions, enabling open‑vocabulary segmentation via natural‑language queries. By subtracting a shared scene direction, whitening text prototypes, and applying soft clustering with CSLS, the method reduces directional concentration and improves class‑relative feature separation. Across seven lifting methods and four datasets, this calibration consistently boosts mean intersection over union for both multiclass and single‑query segmentation without requiring retraining or re‑encoding.
arXiv Computer Vision
Sep 21
A Scene Language Model for Open-Vocabulary Scene Mapping
The paper introduces SceneLM, a vision‑language model that maintains an open‑vocabulary 3D scene map using only a structured text list of objects as persistent memory. The model updates this textual map by adding, editing, and removing objects for each input image, learning the process through supervision tasks and an automatic annotation pipeline. Evaluations on language‑grounded retrieval and localization benchmarks show competitive performance with traditional mapping systems while producing a 6‑12× more compact representation, and the model can run online on an edge device such as a quadruped robot.
Hugging Face Trending Papers
Jul 30
MonoVoc: Decoupling Geometry and Semantics for Lightweight Monocular Open-Vocabulary 3D Gaussians
Open vocabulary 3D scene understanding is essential for next-generation interactive systems, empowering users to intuitively query and navigate reconstructed environments using natural language. However, current 3D Gaussian frameworks are often bottlenecked by restrictive multiview capture requirements, costly scene-specific optimization, and the massive memory overhead of storing dense language features.
arXiv AI
Aug 25
ExtrinSplat: Decoupling Geometry and Semantics for Open-Vocabulary Understanding in 3D Gaussian Splatting
ExtrinSplat is a new framework that separates geometry from semantics in 3D Gaussian Splatting scenes. It clusters Gaussians into overlapping 3D object groups and uses a Vision‑Language Model to generate lightweight textual hypotheses, creating an extrinsic index layer that handles complex polysemy. This approach reduces adaptation time from hours to minutes, cuts storage overhead by orders of magnitude, and outperforms existing embedding‑based methods on open‑vocabulary 3D object selection and semantic segmentation benchmarks.
arXiv Computer Vision
Oct 21
Scene-R1: Video-Grounded Large Language Models for 3D Scene Reasoning with Lightweight Annotations
arXiv:2506.17545v2 Announce Type: replace Abstract: Large language models are increasingly being applied to 3D scene understanding. However, existing 3D-aware LLMs largely operate as black boxes: the...
arXiv AI
Jun 19
QueryGaussian: Scalable and Training-Free Open-Vocabulary 3D Instance Retrieval
arXiv:2606. 19733v1 Announce Type: cross Abstract: Efficiently retrieving specific 3D instances from large-scale scenes via natural language prompts remains a formidable challenge in multimedia analysis.
arXiv AI
Jul 8
OpenGround: Planning-based Online Perception for Open-World 3D Visual Grounding
arXiv:2512. 23020v3 Announce Type: replace-cross Abstract: 3D visual grounding aims to locate objects based on natural language descriptions in 3D scenes.
arXiv AI
2d ago
Generalized Decoupled Learning for Enhancing Open-Vocabulary Dense Perception
arXiv:2508.11256v2 Announce Type: replace-cross Abstract: Dense visual perception tasks have been constrained by their reliance on predefined categories, limiting their applicability in real-world sc...