arXiv Computer Vision

Query-Time Calibration of CLIP Features for Open-Vocabulary 3D Segmentation

The paper introduces a query‑time calibration technique for CLIP features lifted into 3D Gaussian reconstructions, enabling open‑vocabulary segmentation via natural‑language queries. By subtracting a shared scene direction, whitening text prototypes, and applying soft clustering with CSLS, the method reduces directional concentration and improves class‑relative feature separation. Across seven lifting methods and four datasets, this calibration consistently boosts mean intersection over union for both multiclass and single‑query segmentation without requiring retraining or re‑encoding.

arXiv AI

Sep 10

GoDeep: Annotation-Free Open-Vocabulary 3D Scene Understanding via Language-Space Lifting

GoDeep is an annotation‑free method for open‑vocabulary 3D scene understanding that uses a vision‑language model solely as a translator to generate structured, entity‑level descriptions of each image. These descriptions are projected and aggregated in a language‑only embedding space, eliminating the need for a 3D training corpus or domain‑specific encoder. The approach achieves competitive performance on ScanNet++ and a cultural heritage benchmark, accurately localizes out‑of‑vocabulary objects, and offers explainable, point‑level predictions.

By Thodoris Betsas, Anastasios Doulamis, Andreas Georgopoulos

llms rag computer-vision multimodal benchmarks

arXiv AI

Aug 13

CORE-3D: Context-aware Open-vocabulary Retrieval by Embeddings in 3D

arXiv:2509. 24528v4 Announce Type: replace-cross Abstract: Object retrieval from a scene has become a new trend of research due to its numerous applications.

By Mohamad Amin Mirzaei, Pantea Amoie, Ali Ekhterachian, Matin Mirzababaei, Babak Khalaj

llms rag computer-vision multimodal benchmarks

arXiv AI

Jun 19

QueryGaussian: Scalable and Training-Free Open-Vocabulary 3D Instance Retrieval

arXiv:2606. 19733v1 Announce Type: cross Abstract: Efficiently retrieving specific 3D instances from large-scale scenes via natural language prompts remains a formidable challenge in multimedia analysis.

By Xiuyuan Zhu, Ke Lu, Zijie Yang, Chao Yue, Jian Xue, Dongming Zhang

rag computer-vision efficiency benchmarks

arXiv Computer Vision

Oct 8

RT-DETR-World: Transferring Rich LLM Semantics to Real-Time Open-Vocabulary Detection

arXiv:2610.09502v1 Announce Type: new Abstract: Open-vocabulary detection (OVD) recognizes categories unseen during training through textual category queries, yet achieving strong generalization with...

By Yupeng Zhang, Ziyi Zhao, Juntao Cheng, Sheng Wang, Ningnan Guo, Ruize Han, Liang Wan

llms safety

Hugging Face Trending Papers

Oct 7

RT-DETR-World: Transferring Rich LLM Semantics to Real-Time Open-Vocabulary Detection

Open-vocabulary detection (OVD) recognizes categories unseen during training through textual category queries, yet achieving strong generalization with real-time efficiency remains challenging. Beyond...

llms safety

arXiv Computer Vision

Sep 17

NormLift: From Lifted Features To Semantic Reliability In 3D Gaussian Splatting

arXiv:2609.18898v1 Announce Type: new Abstract: Training-free weighted aggregation is widely used to lift 2D semantic features onto 3D Gaussians for open-vocabulary scene understanding, yet its theor...

By Yihan Zang, Da Li, Dominik Engel, Shinkyu Park, Ivan Viola

rag computer-vision safety

arXiv AI

Aug 25

ExtrinSplat: Decoupling Geometry and Semantics for Open-Vocabulary Understanding in 3D Gaussian Splatting

ExtrinSplat is a new framework that separates geometry from semantics in 3D Gaussian Splatting scenes. It clusters Gaussians into overlapping 3D object groups and uses a Vision‑Language Model to generate lightweight textual hypotheses, creating an extrinsic index layer that handles complex polysemy. This approach reduces adaptation time from hours to minutes, cuts storage overhead by orders of magnitude, and outperforms existing embedding‑based methods on open‑vocabulary 3D object selection and semantic segmentation benchmarks.

By Jiayu Ding, Xinpeng Liu, Zhiyi Pan, Shiqiang Long, Ge Li

llms rag computer-vision multimodal benchmarks

arXiv Computer Vision

Sep 18

Scene-Q: Confidence-Aware Coarse-to-Fine Querying of 3D Scenes with Selective VLM Reasoning

Scene-Q is a confidence‑aware, coarse‑to‑fine querying framework for indoor mobile robots that grounds natural‑language queries in a 3D map. It normalizes encoder scores with temperature scaling and only invokes a reasoning VLM for low‑confidence cases, while high‑confidence queries are answered by fast retrieval. The method improves open‑vocabulary 3D instance segmentation on ScanNet200 and natural‑language 3D instance retrieval on real‑world reconstructions, especially for spatial and relational queries, while maintaining a substantial fraction of queries on the fast path.

By Juno Kim, Yesol Park, Hye-Jung Yoon, Byoung-Tak Zhang

computer-vision robotics multimodal

Hugging Face Trending Papers

Jul 30

MonoVoc: Decoupling Geometry and Semantics for Lightweight Monocular Open-Vocabulary 3D Gaussians

Open vocabulary 3D scene understanding is essential for next-generation interactive systems, empowering users to intuitively query and navigate reconstructed environments using natural language. However, current 3D Gaussian frameworks are often bottlenecked by restrictive multiview capture requirements, costly scene-specific optimization, and the massive memory overhead of storing dense language features.

rag computer-vision nlp benchmarks

arXiv Machine Learning

Jul 20

Rethinking the Global Knowledge of CLIP in Training-Free Open-Vocabulary Semantic Segmentation

arXiv:2502. 06818v4 Announce Type: replace Abstract: Recent works modify CLIP to perform open-vocabulary semantic segmentation in a training-free manner (TF-OVSS).

By Jingyun Wang, Cilin Yan, Guoliang Kang

rag computer-vision benchmarks

arXiv Computer Vision

Sep 3

Query Rewriting for Complex Object Segmentation in 4D Gaussian Representations

The paper examines how rewriting verbose, narrative-style queries into concise keyword‑grounded forms improves complex object segmentation in 4D Gaussian representations. By applying a training‑free reinterpretation strategy, the authors reduce linguistic noise while preserving essential semantic anchors. Experiments on HyperNeRF and Neu3D show that rewritten queries boost temporal accuracy from 60.92% to 92.21% and vIoU from 20.08% to 76.94%, with ablation studies confirming the benefits of shorter, keyword‑focused queries.

By Thanh-Khoi Nguyen, Thien-Phuc Tran, Minh-Triet Tran

llms rag computer-vision fine-tuning benchmarks safety

Hugging Face Trending Papers

Jul 21

ZeroSplat: Generalized Referring Segmentation in 3D Gaussian Splatting

Recent advancements in 3D Gaussian Splatting (3DGS) have enabled language-guided scene understanding. However, existing Referring 3D Gaussian Splatting (R3DGS) methods are fundamentally restricted to single-target queries.

llms computer-vision multimodal benchmarks

Related stories