arXiv AI
3d ago

What Should a Navigator Remember? Spatiotemporal Memory for Zero-Shot Vision-Language Navigation

The paper investigates how an 8‑billion‑parameter vision‑language navigation system should store and organize past observations for zero‑shot navigation. It proposes a structured memory comprising task progress, hierarchically compressed observation history tied to poses, and accumulated geometry, which an observer summarizes and a reasoner uses to select waypoint intents or request new views. Experiments on R2R‑CE and real‑world Unitree Go2 robots show that preserving temporal and spatial memories improves success rates and path efficiency compared to baselines that omit these components.

By Yuhang Peng, Zhikai Wang, Ziyi Wang, Jiaru Zhong, Xiaoji Zheng, Jingzhi Huang, Yi Wang, Jiangtao Gong
arXiv Computer Vision
6d ago

Goal2Pixel: Grounding Goals to Pixels for Vision-Language Navigation

Goal2Pixel introduces a pixel‑based approach for vision‑language navigation in continuous environments, replacing low‑level action prediction with navigable pixel grounding. The model predicts a visible pixel on the image plane, which is back‑projected into a 3D waypoint for forward motion, while auxiliary regions handle turning and stopping. Coupled with a visibility‑aware keyframe memory (ViKeyMem) and semantic embeddings, Goal2Pixel achieves state‑of‑the‑art performance on R2R‑CE and RxR‑CE with significantly fewer VLM calls.

By Muyi Bao, Yuxin Cai, Hang Xu, Zongtai Li, Jinxi He, Jingfan Tang, Chen Lv, Ji Zhang, Yaqi Xie, Wenshan Wang