NavHarness: Adaptive Goals for Agentic Vision-Language Navigation
Read the original on arXiv Computer Vision →The Flow has not summarised this story yet — read it at arXiv Computer Vision.
The Flow has not summarised this story yet — read it at arXiv Computer Vision.
The paper introduces NavMCP, a scaffolding framework that couples vision‑language models (VLMs) with navigation foundation models (NFMs) to enable long‑horizon physical‑world agents. NavMCP orchestrates three communication channels—intent, observation, and memory—to allow the VLM to decide what evidence to seek and the NFM to ground semantic sub‑goals into closed‑loop navigation, without retraining either model. The approach achieves state‑of‑the‑art results on several embodied question‑answering benchmarks and significantly outperforms episodic interfaces on the Unitree Go2 robot as task horizons lengthen.
arXiv:2608. 12308v1 Announce Type: cross Abstract: Aerial vision-language navigation (VLN) requires an embodied agent to integrate visual evidence over time, plan future actions, and determine when it has reached a navigation goal under partial observability.
arXiv:2607. 01754v1 Announce Type: new Abstract: On-policy exploration is a crucial component for training robust Vision-Language Navigation agents, as it exposes the policy to a broader state distribution.
NavGPT-3 is a hierarchical navigation runtime that links a large language model trained with long‑horizon reinforcement learning to a low‑latency action policy, enabling an embodied agent to reason, act, and monitor in parallel threads. The system’s OS‑like scheduler selects which thread controls the robot, allowing rapid interruption and reaction to real‑world events. With the NavGPT VLA action policy, the complete harness achieves state‑of‑the‑art performance on R2R‑CE and matches human performance on RxR‑CE, demonstrating a significant advance in autonomous navigation.
On-policy exploration is a crucial component for training robust Vision-Language Navigation agents, as it exposes the policy to a broader state distribution. However, such exploration inevitably leads to trajectories that deviate from expert demonstrations, resulting in a semantic mismatch between the executed visual stream and the original language instruction.
Attacca is a new approach for training visual goal‑conditioned policies that can handle continuous long‑horizon task execution in embodied agents. It decouples goal images from the execution environment by sampling class‑compatible masked goals from other worlds, enabling the policy to ground targets in the current view via a target‑mask prediction head. The method also introduces behavioral‑phase conditioning to distinguish Search, Approach, and Interact stages, and achieves significant performance gains on Minecraft tasks, outperforming baselines by up to 7× on long‑horizon scenarios.