arXiv Machine Learning

Less Language, More Latents: Annotation-Efficient VLAs for Driving

The paper introduces Latent Action Driving Annotations (LADA), a three‑stage pipeline that converts large amounts of unlabelled observation‑trajectory data into a language‑conditioned driving model. First, a latent action model with a vector‑quantised bottleneck learns a compact codebook of vehicle intents. Then, a small set of language‑annotated examples trains a vision‑language translator to map observations and instructions into this codebook, and finally a VLA is trained on observation‑latent‑action pairs across the full corpus. Using less than 5% of language annotations, LADA attains a Driving Score of 87.98 and a Success Rate of 70.46% on Bench2Drive, matching or surpassing fully supervised baselines.

arXiv Computer Vision
Oct 1

Vision-Language-Action Autonomous Driving Agent with Language-based Memory

arXiv:2609.38641v1 Announce Type: new Abstract: Vision-Language-Action (VLA) foundation models have recently emerged as one of the prevailing solutions for autonomous driving, as they can utilize kno...

By Kai Yan, Xiangyu Chen, Yulong Cao, Alex Naumann, Peter Karkus, Yan Wang, Jef Packer, Alex Schwing, Yuxiong Wang, Boris Ivanovic, Wenjie Luo, Marco Pavone
arXiv Computer Vision
Sep 4

Continuous Actions from Discrete Minds: Latent-Aligned Planning for End-to-End Autonomous Driving

The paper introduces LaPla, a Vision‑Language‑Action framework that uses a latent‑aligned planning approach to convert discrete semantic reasoning into continuous, physics‑constrained driving actions. It employs a residual VQ‑VAE to encode vehicle kinematics into a structured latent space, then projects multimodal inputs—images, past actions, and text—directly into this latent space, allowing a frozen decoder to generate physically plausible trajectories without quantization errors. Experiments on nuScenes and NVIDIA AlpaSim show LaPla reduces long‑horizon L2 error by 15.52% and improves closed‑loop success rates by 33.34 percentage points while cutting inference latency.

By Ruoyu Yao, Yusen Xie, Qingzhao Liu, Pei Liu, Zewei Yang, Yipeng Zhu, Xiaolong Wang, Jun Ma
arXiv AI
Oct 15

DriveWLA: Unifying Future Dynamics and Scene Semantics through World-Language-Action Modeling for Autonomous Driving

DriveWLA is a unified World‑Language‑Action model that integrates semantic understanding, future scene dynamics, and trajectory generation for autonomous driving. It employs a Mixture‑of‑Transformers architecture with three experts—World, Language, and Action—connected by Cross‑Expert Attention to align semantic and dynamic representations. After reinforcement learning refinement, DriveWLA outperforms existing VLM‑based and world‑model‑based planners on NAVSIM benchmarks and generalizes well to other datasets without fine‑tuning.

By Shuzheng Wang, Haoxuan Li, Shengzhen Li, Chenxi Wu, Yajing Sun, Lu Hou, Wenyu Liu, Xinggang Wang
arXiv Computer Vision
Oct 21

DriveVLA-M0: Failure-Aware Memory Augmentation for Autonomous Driving

DriveVLA-M0 introduces a failure-aware latent memory system for Vision‑Language‑Action models in autonomous driving. It stores past failure cases with scene representations and expert trajectories, and uses a retrieval model to decouple static road structure from dynamic agent interactions. During inference, retrieved cases are injected via a lightweight LoRA‑based test‑time training mechanism, yielding significant performance gains on NAVSIM benchmarks with minimal latency overhead.

By Zebin Xing, Yupeng Zheng, Qiang Chen, Linbo Wang, Yichen Zhang, Pengxuan Yang, Junli Wang, Deheng Qian, Xiaoqing Ye, Junyu Han, Yifeng Pan, Qichao Zhang, Dongbin Zhao
arXiv Computer Vision
6d ago

Generative Scenario Rollouts for End-to-End Autonomous Driving

Generative Scenario Rollouts (GeRo) is a plug‑and‑play framework that augments Vision‑Language‑Action models for autonomous driving by jointly planning and generating language‑grounded future traffic scenes through an autoregressive rollout strategy. The approach trains a VLA model to encode vehicle and agent dynamics into latent tokens, then uses language‑conditioned autoregressive generation to produce future latent tokens and textual responses, with a rollout‑consistency loss to maintain temporal coherence and text‑action alignment. On the Bench2Drive benchmark, GeRo improves driving score by +15.7 and success rate by +26.2, and achieves state‑of‑the‑art closed‑loop and open‑loop performance, demonstrating strong zero‑shot robustness.

By Rajeev Yasarla, Deepti Hegde, Shizhong Han, Hsin-Pai Cheng, Yunxiao Shi, Meysam Sadeghigooghari, Shweta Mahajan, Apratim Bhattacharyya, Litian Liu, Risheek Garrepalli, Thomas Svantesson, Fatih Porikli, Mohammad Ghavamzadeh, Hong Cai
arXiv Computer Vision
Sep 3

LookStep: Efficient Vision-Language Navigation with Linguistic Foresight and Event Driven Memory

LookStep is a new end‑to‑end framework for Vision‑Language Navigation that integrates Language‑Centric Future State Modeling with an Event‑Driven Rolling Memory. It uses language labels to predict coarse navigation progress and future states for candidate actions, and autonomously decides which observations to store in a bounded memory with semantic roles. Empirical results show that LookStep outperforms existing methods on VLN‑CE tasks, achieving a 49.7% success rate on R2R‑CE Val‑Unseen while improving memory efficiency and reducing data requirements.

By Kun-Yang Yu, Yingzhe Li, Hongyu Xu, Shi-Yu Tian, Zhi Zhou, Yang Chen, Ming Yang, Sheng Wang, Qing Yu, Lan-Zhe Guo, Yu-Feng Li
arXiv Computer Vision
Oct 27

AdaThinkDrive: Adaptive Thinking via Reinforcement Learning for Autonomous Driving

arXiv:2509.13769v2 Announce Type: replace Abstract: While reasoning technology like Chain of Thought (CoT) has been widely adopted in Vision Language Action (VLA) models, it demonstrates promising ca...

By Yuechen Luo, Fang Li, Shaoqing Xu, Zhiyi Lai, Lei Yang, Qimao Chen, Ziang Luo, Zixun Xie, Shengyin Jiang, Jiaxin Liu, Long Chen, Bing Wang, Ken Deng, Zhi-xin Yang
arXiv Computer Vision
Aug 27

Latent Chain-of-Thought World Modeling for End-to-End Driving

Latent-CoT-Drive (LCDrive) is a vision‑language‑action model for autonomous driving that replaces natural‑language chain‑of‑thought reasoning with a latent language capturing possible outcomes of driving actions. The model interleaves action‑proposal tokens, aligned with the model’s output actions, and world‑model tokens grounded in a learned latent world model to reason about future outcomes. After a supervised cold‑start using ground‑truth future rollouts, LCDrive is further refined with closed‑loop reinforcement learning, achieving faster inference, higher‑quality trajectories, and greater gains from interactive RL than both non‑reasoning and text‑reasoning baselines on a large‑scale end‑to‑end driving benchmark.

By Shuhan Tan, Kashyap Chitta, Yuxiao Chen, Ran Tian, Yurong You, Yan Wang, Wenjie Luo, Yulong Cao, Philipp Krahenbuhl, Marco Pavone, Boris Ivanovic
arXiv AI
Jul 9

VOTE: Vision-Language-Action Optimization with Trajectory Ensemble Voting

arXiv:2507. 05116v5 Announce Type: replace-cross Abstract: Recent large-scale Vision Language Action (VLA) models have shown superior performance in robotic manipulation tasks guided by natural language.

By Juyi Lin, Amir Taherin, Arash Akbari, Arman Akbari, Lei Lu, Guangyu Chen, Taskin Padir, Xiaomeng Yang, Weiwei Chen, Yiqian Li, Xue Lin, David Kaeli, Pu Zhao, Yanzhi Wang