arXiv Computer Vision

NavHarness: Adaptive Goals for Agentic Vision-Language Navigation

arXiv AI
Sep 1

Scaffolding Foundation Models into Physical-World Agents Pushes the Frontier of Long-Horizon Navigation

The paper introduces NavMCP, a scaffolding framework that couples vision‑language models (VLMs) with navigation foundation models (NFMs) to enable long‑horizon physical‑world agents. NavMCP orchestrates three communication channels—intent, observation, and memory—to allow the VLM to decide what evidence to seek and the NFM to ground semantic sub‑goals into closed‑loop navigation, without retraining either model. The approach achieves state‑of‑the‑art results on several embodied question‑answering benchmarks and significantly outperforms episodic interfaces on the Unitree Go2 robot as task horizons lengthen.

By Zixing Lei, Gengze Zhou, Xiong-Hui Chen, Jiazhao Zhang, Yiyang Huang, Hang Yin, Haoqi Yuan, Qi Wu, Weixin Li, Siheng Chen
arXiv AI
Oct 9

NavGPT-3: Harnessing Context in a Hierarchical Navigation Runtime

NavGPT-3 is a hierarchical navigation runtime that links a large language model trained with long‑horizon reinforcement learning to a low‑latency action policy, enabling an embodied agent to reason, act, and monitor in parallel threads. The system’s OS‑like scheduler selects which thread controls the robot, allowing rapid interruption and reaction to real‑world events. With the NavGPT VLA action policy, the complete harness achieves state‑of‑the‑art performance on R2R‑CE and matches human performance on RxR‑CE, demonstrating a significant advance in autonomous navigation.

By Gengze Zhou, Yicong Hong, Jiazhao Zhang, Xunyi Zhao, Jian Zhou, Zixing Lei, Zun Wang, Chongyang Zhao, Xionghui Chen, Stephen Gould, Anton van den Hengel, Qi Wu
Hugging Face Trending Papers
Jul 2

Path-level Hindsight Instructions for Semantic Exploration in Vision-Language Navigation

On-policy exploration is a crucial component for training robust Vision-Language Navigation agents, as it exposes the policy to a broader state distribution. However, such exploration inevitably leads to trajectories that deviate from expert demonstrations, resulting in a semantic mismatch between the executed visual stream and the original language instruction.

arXiv AI
Oct 7

Attacca: Goal-Directed Control under State Continuity for Long-Horizon Embodied Agents

Attacca is a new approach for training visual goal‑conditioned policies that can handle continuous long‑horizon task execution in embodied agents. It decouples goal images from the execution environment by sampling class‑compatible masked goals from other worlds, enabling the policy to ground targets in the current view via a target‑mask prediction head. The method also introduces behavioral‑phase conditioning to distinguish Search, Approach, and Interact stages, and achieves significant performance gains on Minecraft tasks, outperforming baselines by up to 7× on long‑horizon scenarios.

By Gyusik Seo, Jaehong Yoon
arXiv Computer Vision
Sep 22

What do VLM-Based Vision-Language Navigation Models Rely on: Interpreting and Steering Policy Behavior

arXiv:2609.24576v1 Announce Type: cross Abstract: Modern Vision-Language Navigation (VLN) models rely mostly on pre-trained large Vision-Language Models (VLMs) to predict navigation actions. While th...

By D\'ebora Oliveira Makowski, Samiran Gode, Abhijeet Nayak, Marco Hutter, Cordelia Schmid, Lukas Rosenberger Schmid, Wolfram Burgard
arXiv AI
Oct 31

What Should a Navigator Remember? Spatiotemporal Memory for Zero-Shot Vision-Language Navigation

The paper investigates how an 8‑billion‑parameter vision‑language navigation system should store and organize past observations for zero‑shot navigation. It proposes a structured memory comprising task progress, hierarchically compressed observation history tied to poses, and accumulated geometry, which an observer summarizes and a reasoner uses to select waypoint intents or request new views. Experiments on R2R‑CE and real‑world Unitree Go2 robots show that preserving temporal and spatial memories improves success rates and path efficiency compared to baselines that omit these components.

By Yuhang Peng, Zhikai Wang, Ziyi Wang, Jiaru Zhong, Xiaoji Zheng, Jingzhi Huang, Yi Wang, Jiangtao Gong
arXiv Computer Vision
Oct 9

SuperNav: An Agentic Navigation System for Any Task in Any Scene

SuperNav is a navigation system that leverages a pretrained multimodal large language model (MLLM) without fine‑tuning it for navigation. The system introduces an agent harness that delegates motion execution to navigation tools while the MLLM focuses on interpreting requests, understanding scenes, and making decisions. A visual‑point interface connects the model’s decisions to motion, allowing direct destination specification and feedback‑driven revision, enabling sustained navigation across varied tasks and environments.

By Jinkai Zhang, Jingyi Xu, Yuanhong Yu, Jiarui Guo, Ruizhen Hu, Hujun Bao, Xiaowei Zhou, Sida Peng