arXiv Computer Vision By Kangan Qian, ChuChu Xie, Yang Zhong, Jingrui Pang, Siwen Jiao, Sicong Jiang, Zilin Huang, Yunlong Wang, Kun Jiang, Mengmeng Yang, Hao Ye, Guanghao Zhang, Hangjun Ye, Guang Chen, Long Chen, Diange Yang

XEmbodied: A Foundation Model with Enhanced Geometric and Physical Cues for Large-Scale Embodied Environments

Read the original on arXiv Computer Vision →

The Flow has not summarised this story yet — read it at arXiv Computer Vision.

arXiv AI
Sep 2

Towards Generalizable Visually Grounded Exploration of Household Devices

The paper introduces VGEBench, a new benchmark for evaluating Vision‑Language Models (VLMs) on generalizable, visually grounded exploration of household devices. Unlike existing datasets that rely on static images or annotated trajectories, VGEBench employs a logic‑driven state machine to simulate multi‑turn interaction loops, requiring agents to actively perceive, act, and refine their actions to achieve goals. Experiments show that current VLMs struggle to translate semantic knowledge into physical execution and to maintain long‑horizon state tracking.

By Linhao Zheng, Zeming Liu, Wangke Chen, Li Zeng, Wanxiang Che, Heyan Huang, Yuhang Guo
arXiv Computer Vision
Sep 23

Metric-Bench: Exploring In-context Spatial Metric Reasoning in VLMs for Indoor Scenes

Metric-Bench introduces a new benchmark for Vision‑Language Models (VLMs) that focuses on metric‑spatial reasoning in indoor scenes by using in‑image reference objects with known dimensions. The accompanying MetricReasoner fine‑tuning recipe employs structured prompts and numerical rewards to implicitly learn 2D‑to‑3D mapping without camera intrinsics. Experiments show that this approach improves spatial metric understanding by 43.1 % over existing models and boosts downstream embodied tasks, while also delivering gains on general VLM benchmarks.

By Yuling Xi, Haokai Zhang, Muzhi Zhu, Hao Zhong, Zongze Du, Hengyu Zhao, Chenchen Jing, Yufei Yin, Bin Qin, Yongjie Yang, Zhenbo Luo, Hao Chen, Chunhua Shen
arXiv AI
3d ago

From Foundation VLM to Versatile Driving VLM: A Triplet Incremental Training Approach

The paper introduces the Triplet Incremental Training Framework, a non‑intrusive method that adds 3D geometric awareness to general Vision‑Language Models (VLMs) for autonomous driving. By training on automatically generated (image, depth, QA) triplets, the framework uses an auxiliary depth stream to guide the VLM’s visual backbone while a frozen twin stream prevents semantic drift. The depth branch is removed at inference, yielding zero overhead, and the resulting model outperforms state‑of‑the‑art driving VLMs while preserving general reasoning abilities.

By Shengtao Wu, Zichao Dong, Weixuan Qiao, Xiaolu Liu, Xin Zhan, Junbo Chen, Jianke Zhu
arXiv Computer Vision
Sep 15

Stereo4DWalker: Learning 4D-aware Embodied Urban Navigation from Internet Stereo Videos

Stereo4DWalker is a 4D-aware embodied navigation model that uses stereo video inputs to explicitly construct structured representations of geometry and motion. These 4D structures are incorporated into a navigation transformer via 4D-conditioned attention layers, enabling the agent to learn robust urban navigation. The authors also curate a large-scale stereo navigation dataset with automatically annotated actions from Internet stereo videos, and demonstrate that Stereo4DWalker outperforms state‑of‑the‑art methods while requiring only 1.5% of the training data.

By Wentao Zhou, Xuweiyi Chen, Vignesh Rajagopal, Jeffrey Chen, Rohan Chandra, Zezhou Cheng