arXiv Computer Vision

From Alignment to Fusion in 3D Vision-Language

arXiv Computer Vision
6d ago

Multi-View Relational Distillation for Spatial Reasoning with Vision-Language Models

The paper introduces Multi-View Relational Distillation (MVRD), a method that distills patch-wise cosine similarities across multiple views from geometry‑grounded vision models into vision‑language models (VLMs). By focusing on relational geometry rather than raw teacher features, MVRD preserves the pretrained alignment between visual and textual representations while enhancing spatial reasoning. Experiments show that MVRD outperforms supervised fine‑tuning and feature distillation, approaching feature‑fusion performance with fewer parameters and lower latency, and it generalizes to 3D scene understanding tasks such as object grounding, dense captioning, and question answering.

By Kiet T. Nguyen, Hanbo Shim, Jinwoo Kim, Seunghoon Hong
arXiv Computer Vision
Oct 28

3D-SceneQ: Empowering 3D LLM With Query-Guided Adaptive Pruning and Multi-Modal Feature Enhancement

3D-SceneQ is a new 3D large language model that tackles two key limitations of existing 3D scene understanding pipelines: the inclusion of irrelevant object tokens that add noise, and the exclusive use of point cloud data that lacks rich semantic cues. It introduces a Query‑Guided Adaptive Pruning (QGAP) module that selectively keeps task‑relevant objects based on user instructions, and a Multi‑modal Object‑level Feature Enhancement (MOFE) module that fuses 2D feature embeddings into the 3D representation to enrich semantic information. Experiments show that 3D‑SceneQ surpasses state‑of‑the‑art methods on several benchmarks, achieving significant gains in metrics such as CIDEr and EM@1.

By Yongqi Shan, Lu Zhang, Jiazuo Yu, Yunzhi Zhuge, Huchuan Lu
arXiv Computation and Language
Sep 30

Imagine3D-LLM: Teaching MLLMs to Imagine 3D Scenes Before Answering

arXiv:2609.38177v1 Announce Type: cross Abstract: Reasoning about the 3D world from multi-view images remains a fundamental challenge for Multimodal Large Language Models (MLLMs). While modern MLLMs...

By Jaewoo Jung, Hyeonseo Yu, Honggyu An, Jisang Han, Mungyeom Kim, Minkyeong Jeon, Heeseong Shin, Wonjun Moon, Federico Tombari, Daniel Barath, Marc Pollefeys, Seungryong Kim, Sunghwan Hong
arXiv AI
Sep 10

MV-STRIDE: Enabling MLLMs to Master Multi-View Spatial Reasoning via Hierarchical Capability Modeling

MV-STRIDE is a Multi‑View hierarchical Spatial Reasoning dataset that models dependencies among perception, scene understanding, and contextual reasoning to support 3D spatial cognition. It introduces a QA generation pipeline that enforces cross‑view constraints, producing multi‑level reasoning tasks with chain‑of‑thought supervision. Experiments show that training on MV‑STRIDE yields state‑of‑the‑art performance on multi‑view spatial benchmarks, enabling MLLMs to reason robustly across diverse viewpoints.

By Jin Xu, Xiaojian Huang, Zhuodong Luo, Zhihong Zhang, Xin Liu, Jiansheng Wei, Xinzhi Wang, Jie Zhao, Xuejin Chen