arXiv AI By Shengtao Wu, Zichao Dong, Weixuan Qiao, Xiaolu Liu, Xin Zhan, Junbo Chen, Jianke Zhu

From Foundation VLM to Versatile Driving VLM: A Triplet Incremental Training Approach

Read the original on arXiv AI →

The paper introduces the Triplet Incremental Training Framework, a non‑intrusive method that adds 3D geometric awareness to general Vision‑Language Models (VLMs) for autonomous driving. By training on automatically generated (image, depth, QA) triplets, the framework uses an auxiliary depth stream to guide the VLM’s visual backbone while a frozen twin stream prevents semantic drift. The depth branch is removed at inference, yielding zero overhead, and the resulting model outperforms state‑of‑the‑art driving VLMs while preserving general reasoning abilities.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

arXiv AI
Jul 24

3D-Aware VLMs with Implicit and Explicit Geometries

arXiv:2607. 21595v1 Announce Type: cross Abstract: Despite rapid progress, most existing vision-language models (VLMs) built from 2D visual inputs often struggle when handling various 3D tasks that require fine-grained spatial understanding and reasoning.

By Wenhao Li, Xueying Jiang, Quanhao Qian, Deli Zhao, Ran Xu, Shijian Lu, Gongjie Zhang
arXiv AI
Oct 2

Task-Adaptive Grounded 3D-Programmers Using 2D VLMs

The paper introduces 3D-Prog, a framework that adapts powerful 2D vision‑language models (VLMs) for reliable 3D understanding, manipulation, and generation. It does so by employing Canonical Coordinate Framing (CCF) to anchor inputs and outputs in a shared Euclidean coordinate system, resolving axis ambiguity, metric scale, and reference issues, and Task‑Adaptive Feedback (TAF) to close the reasoning loop with dynamic, task‑specific feedback. Together, CCF and TAF enable 2D VLMs to perform open‑vocabulary 3D tasks without retraining, yielding consistent, interpretable, and high‑quality results across diverse 3D scenarios.

By Arman Raayatsanati, Sombit Dey, Anna-Maria Halacheva, Jan-Nico Zaech, Luc Van Gool, Danda Pani Paudel
arXiv Computer Vision
Oct 21

HERMES++: Toward a Unified Driving World Model for 3D Scene Understanding and Generation

HERMES++ is a unified driving world model that simultaneously performs 3D scene understanding and future geometry prediction. It uses a BEV representation to merge multi‑view spatial data for LLM compatibility, introduces LLM‑enhanced world queries for knowledge transfer, and employs a Current‑to‑Future Link to condition future geometry on semantic context. A Joint Geometric Optimization strategy enforces structural integrity by combining explicit geometric constraints with implicit latent regularization, and extensive benchmark tests show that HERMES++ outperforms specialist methods in both point cloud prediction and scene understanding.

By Xin Zhou, Dingkang Liang, Xiwu Chen, Feiyang Tan, Dingyuan Zhang, Hengshuang Zhao, Xiang Bai
arXiv AI
Sep 10

MV-STRIDE: Enabling MLLMs to Master Multi-View Spatial Reasoning via Hierarchical Capability Modeling

MV-STRIDE is a Multi‑View hierarchical Spatial Reasoning dataset that models dependencies among perception, scene understanding, and contextual reasoning to support 3D spatial cognition. It introduces a QA generation pipeline that enforces cross‑view constraints, producing multi‑level reasoning tasks with chain‑of‑thought supervision. Experiments show that training on MV‑STRIDE yields state‑of‑the‑art performance on multi‑view spatial benchmarks, enabling MLLMs to reason robustly across diverse viewpoints.

By Jin Xu, Xiaojian Huang, Zhuodong Luo, Zhihong Zhang, Xin Liu, Jiansheng Wei, Xinzhi Wang, Jie Zhao, Xuejin Chen