← Back to all news
arXiv Computer Vision By Kun-Yang Yu, Yingzhe Li, Hongyu Xu, Shi-Yu Tian, Zhi Zhou, Yang Chen, Ming Yang, Sheng Wang, Qing Yu, Lan-Zhe Guo, Yu-Feng Li

LookStep: Efficient Vision-Language Navigation with Linguistic Foresight and Event Driven Memory

Read the original on arXiv Computer Vision →

LookStep is a new end‑to‑end framework for Vision‑Language Navigation that integrates Language‑Centric Future State Modeling with an Event‑Driven Rolling Memory. It uses language labels to predict coarse navigation progress and future states for candidate actions, and autonomously decides which observations to store in a bounded memory with semantic roles. Empirical results show that LookStep outperforms existing methods on VLN‑CE tasks, achieving a 49.7% success rate on R2R‑CE Val‑Unseen while improving memory efficiency and reducing data requirements.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computer Vision.

Related stories

arXiv Computer Vision
Oct 23

UniMem: Unifying Visual and Textual Memory in VLAs for Non-Markovian Tasks

UniMem is a framework that unifies visual and textual memory with low‑level control in a single Vision‑Language‑Action model, using an event classifier, keyframe encoder, and caching to manage memory efficiently. It addresses non‑Markovian tasks that require remembering past scenes, avoiding the memory bottleneck of separate VLMs and the performance drop from fixed‑interval frame sampling. Across five simulation and four hardware tasks, UniMem outperforms fixed‑interval sampling (93.4% vs. 68.2%) and hierarchical VLM‑VLA baselines (80.0% vs. 43.5%) while offering faster inference and a simpler training pipeline.

By Lars Osterberg, Maggie Wang, Mac Schwager
fine-tuningmultimodal
More like this →
arXiv AI
Jul 3

CoFL-S: Spatially Queryable Sector Flow Fields for Local Language-Conditioned Navigation

arXiv:2607. 02222v1 Announce Type: cross Abstract: Vision-Language Navigation has increasingly emphasized high-level instruction reasoning, memory, global map construction, and instruction decomposition, while the low-level action representation remains comparatively underexplored.

By Haokun Liu, Zhaoqi Ma, Yicheng Chen, Wentao Zhang, Masaki Kitagawa, Zicen Xiong, Jinjie Li, Moju Zhao
roboticsmultimodalbenchmarks
More like this →
arXiv Computer Vision
Sep 22

What do VLM-Based Vision-Language Navigation Models Rely on: Interpreting and Steering Policy Behavior

arXiv:2609.24576v1 Announce Type: cross Abstract: Modern Vision-Language Navigation (VLN) models rely mostly on pre-trained large Vision-Language Models (VLMs) to predict navigation actions. While th...

By D\'ebora Oliveira Makowski, Samiran Gode, Abhijeet Nayak, Marco Hutter, Cordelia Schmid, Lukas Rosenberger Schmid, Wolfram Burgard
llmsagentsfine-tuningmultimodalsafety
More like this →
arXiv Machine Learning
Sep 24

Less Language, More Latents: Annotation-Efficient VLAs for Driving

The paper introduces Latent Action Driving Annotations (LADA), a three‑stage pipeline that converts large amounts of unlabelled observation‑trajectory data into a language‑conditioned driving model. First, a latent action model with a vector‑quantised bottleneck learns a compact codebook of vehicle intents. Then, a small set of language‑annotated examples trains a vision‑language translator to map observations and instructions into this codebook, and finally a VLA is trained on observation‑latent‑action pairs across the full corpus. Using less than 5% of language annotations, LADA attains a Driving Score of 87.98 and a Success Rate of 70.46% on Bench2Drive, matching or surpassing fully supervised baselines.

By Alexey Zakharov, Kemal Oksuz, Puneet K. Dokania
agentsnlproboticsmultimodalbenchmarks
More like this →