arXiv Computer Vision

XEmbodied: A Foundation Model with Enhanced Geometric and Physical Cues for Large-Scale Embodied Environments

arXiv AI
Sep 2

Towards Generalizable Visually Grounded Exploration of Household Devices

The paper introduces VGEBench, a new benchmark for evaluating Vision‑Language Models (VLMs) on generalizable, visually grounded exploration of household devices. Unlike existing datasets that rely on static images or annotated trajectories, VGEBench employs a logic‑driven state machine to simulate multi‑turn interaction loops, requiring agents to actively perceive, act, and refine their actions to achieve goals. Experiments show that current VLMs struggle to translate semantic knowledge into physical execution and to maintain long‑horizon state tracking.

By Linhao Zheng, Zeming Liu, Wangke Chen, Li Zeng, Wanxiang Che, Heyan Huang, Yuhang Guo
arXiv Computer Vision
Sep 23

Metric-Bench: Exploring In-context Spatial Metric Reasoning in VLMs for Indoor Scenes

Metric-Bench introduces a new benchmark for Vision‑Language Models (VLMs) that focuses on metric‑spatial reasoning in indoor scenes by using in‑image reference objects with known dimensions. The accompanying MetricReasoner fine‑tuning recipe employs structured prompts and numerical rewards to implicitly learn 2D‑to‑3D mapping without camera intrinsics. Experiments show that this approach improves spatial metric understanding by 43.1 % over existing models and boosts downstream embodied tasks, while also delivering gains on general VLM benchmarks.

By Yuling Xi, Haokai Zhang, Muzhi Zhu, Hao Zhong, Zongze Du, Hengyu Zhao, Chenchen Jing, Yufei Yin, Bin Qin, Yongjie Yang, Zhenbo Luo, Hao Chen, Chunhua Shen
arXiv AI
3d ago

From Foundation VLM to Versatile Driving VLM: A Triplet Incremental Training Approach

The paper introduces the Triplet Incremental Training Framework, a non‑intrusive method that adds 3D geometric awareness to general Vision‑Language Models (VLMs) for autonomous driving. By training on automatically generated (image, depth, QA) triplets, the framework uses an auxiliary depth stream to guide the VLM’s visual backbone while a frozen twin stream prevents semantic drift. The depth branch is removed at inference, yielding zero overhead, and the resulting model outperforms state‑of‑the‑art driving VLMs while preserving general reasoning abilities.

By Shengtao Wu, Zichao Dong, Weixuan Qiao, Xiaolu Liu, Xin Zhan, Junbo Chen, Jianke Zhu
arXiv Computer Vision
Sep 15

Stereo4DWalker: Learning 4D-aware Embodied Urban Navigation from Internet Stereo Videos

Stereo4DWalker is a 4D-aware embodied navigation model that uses stereo video inputs to explicitly construct structured representations of geometry and motion. These 4D structures are incorporated into a navigation transformer via 4D-conditioned attention layers, enabling the agent to learn robust urban navigation. The authors also curate a large-scale stereo navigation dataset with automatically annotated actions from Internet stereo videos, and demonstrate that Stereo4DWalker outperforms state‑of‑the‑art methods while requiring only 1.5% of the training data.

By Wentao Zhou, Xuweiyi Chen, Vignesh Rajagopal, Jeffrey Chen, Rohan Chandra, Zezhou Cheng
arXiv Machine Learning
Jul 8

PoseVLA: Universal Pose Pretraining for Generalizable Vision-Language-Action Policies

arXiv:2602. 19710v3 Announce Type: replace-cross Abstract: Existing Vision-Language-Action (VLA) models often suffer from feature collapse and low training efficiency because they entangle high-level perception with sparse, embodiment-specific action supervision.

By Haitao Lin, Hanyang Yu, Jingshun Huang, He Zhang, Yonggen Ling, Ping Tan, Xiangyang Xue, Yanwei Fu
arXiv Computer Vision
Oct 16

EPIC-Bench: A Perception-Centric Benchmark for Fine-Grained Embodied Visual Grounding in Vision-Language Models

EPIC-Bench is a perception‑centric benchmark for fine‑grained embodied visual grounding in vision‑language models. It contains 6.6k human‑annotated tuples of Image, Text, and Mask across the three core stages of embodied interaction—Target Localization, Navigation, and Manipulation. The benchmark offers 23 fine‑grained tasks, multi‑dimensional settings, and evaluates over 80 leading VLMs to provide diagnostic insights into their vision‑language grounding capabilities.

By Haozhe Shan, Xiancong Ren, Han Dong, Yingji Zhang, Haoyuan Shi, Jiayu Hu, Qinfan Zhang, Yi Zhang, Yong Dai, Bin Shen, Lizhen Qu, Zenglin Xu, Xiaozhu Ju
arXiv AI
Jul 24

3D-Aware VLMs with Implicit and Explicit Geometries

arXiv:2607. 21595v1 Announce Type: cross Abstract: Despite rapid progress, most existing vision-language models (VLMs) built from 2D visual inputs often struggle when handling various 3D tasks that require fine-grained spatial understanding and reasoning.

By Wenhao Li, Xueying Jiang, Quanhao Qian, Deli Zhao, Ran Xu, Shijian Lu, Gongjie Zhang
arXiv AI
Jun 9

GEAR-VLA: Learning Geometry-Aware Action Representations for Generalizable Robotic Manipulation

arXiv:2606. 08530v1 Announce Type: cross Abstract: Vision-Language-Action (VLA) models achieve strong benchmark performance but still struggle in real-world deployment with unseen objects, background shifts, and different robot embodiments.

By Yuan Zhang, Shiqi Zhang, Yedong Shen, Shuai Dong, Jiajun Deng, Xin Zhang, Yuxuan Gao, Jiajia Wu, Xin Nie, Zhiyuan Cheng, Jianmin Ji, Yanyong Zhang, Xingyi Zhang, Jia Pan