SimWAM is a lightweight World-Action Model that uses future‑video prediction only during training to supervise an action expert, enabling end‑to‑end autonomous driving without costly test‑time future imagination. The architecture co‑trains a pretrained video expert and a lightweight action expert via joint flow matching, while an isolated attention mask keeps action prediction independent of future frames. This design allows the video backbone to be swapped and the action expert scaled independently, achieving 91.5 PDMS on NAVSIM, outperforming state‑of‑the‑art WAM planners with lower latency and zero‑shot transfer to nuScenes.
arXiv:2606.27504v2 Announce Type: replace Abstract: World Action Models (WAMs) unify future environment prediction with action generation for autonomous driving, yet existing approaches optimize only...
SimVTA is a post‑training framework that enhances driving world action models by aligning simulated future videos with candidate trajectories. It samples trajectories, renders high‑reward futures in a real‑world simulator using 3D Gaussian Splatting, and prioritizes video supervision for high‑error futures. The approach improves planning performance, raising PDMS from 86.2 to 91.3 on NAVSIM v1 and achieving strong results on NAVSIM v2 and HUGSIM.
arXiv:2608.20974v1 Announce Type: cross Abstract: Video Joint Embedding Predictive Architecture (V-JEPA) learns powerful spatiotemporal representations from video through self-supervised latent featu...
Video Prediction Policy 2 (VPP2) is a new world action model that improves zero‑shot generalization for both video prediction and action generation. It achieves this by pretraining a large, diverse manipulation video dataset with event‑level supervision, then distilling the model into a single‑step visual planner and adding a mixture‑of‑transformers action module. Experiments show VPP2 outperforms leading baselines on open‑ended video prediction, real‑world zero‑shot manipulation, and several challenging benchmarks.
MT‑WAM enhances the Fast‑WAM framework by adding complementary supervision for future 2‑D point trajectories and visual features while keeping the original training objectives. A lightweight dual‑stream branch and structured attention mask isolate motion‑specific processing, and motion‑stream tokens provide additional dynamics cues to the action expert. During inference, MT‑WAM skips future‑video prediction, using cached video and motion information to achieve higher success rates on LIBERO, LIBERO‑Plus, RoboTwin 2.0 Clean2Rand, and several real‑world tasks.
The paper introduces NAVA-WAM, a method that learns action priors directly from observation-only videos to pretrain action policies for robots. It uses a two-stage training process: first, it pretrains on videos with future‑video flow‑matching supervision to learn action‑relevant priors, and second, it fine‑tunes with action‑labeled demonstrations for robot control. Experiments show that NAVA‑WAM outperforms previous methods in both in‑distribution and out‑of‑distribution scenarios, achieving strong action‑label efficiency and real‑robot generalization.
The survey "World-Action Models for Robot Learning and Control" reviews recent advances in coupling future world prediction with executable action generation for robots in open environments. It clarifies the scope of World-Action Models (WAMs) relative to conventional world models, model-based RL, and Vision‑Language‑Action policies, and organizes existing methods through a unified taxonomy covering representations, transition modeling, action interfaces, architectures, training pipelines, data modalities, and scaling strategies. The paper also surveys applications in manipulation, navigation, and autonomous driving, summarizes datasets, benchmarks, and metrics, and discusses key challenges such as action alignment, spatial consistency, long‑horizon memory, and efficient inference.
Autonomous driving requires planning under both semantic constraints and predictive dynamics. Existing end-to-end driving approaches, however, typically emphasize only one side of this requirement: Vision-Language-Action (VLA) models exploit VLM priors for semantic reasoning, while World Action Models (WAMs) provide future-aware prediction through generative world modeling.
CtrlWAM introduces a controllable world action model that jointly predicts actions (intent) and visual futures (foresight). By executing perturbed actions in a simulator and pairing them with noised visual outcomes, it aligns action predictions with their visual consequences, using warped video–action noise schedules to maintain visual layout responsiveness. The model extends beyond ego‑only control to multiple agent streams, improving action forecasts, video–action agreement, and command following in driving and robotics experiments.
arXiv:2608. 12854v1 Announce Type: cross Abstract: Autonomous driving requires planning under both semantic constraints and predictive dynamics.
SV-WAM is a surround‑view world‑action model that keeps all six camera views for autonomous driving while enabling efficient inference by discarding the video branch during deployment. It uses future‑video prediction as dense training supervision and introduces an action‑centered causal mask to prevent future‑video tokens from influencing action tokens during joint denoising. A differentiable drivable‑area compliance regularizer further improves safety by penalizing vehicle‑footprint corners that approach or cross drivable boundaries. Experiments on NAVSIMv2 and nuScenes show state‑of‑the‑art planning performance with low latency and strong zero‑shot transfer.
SV-WAM is a surround‑view world‑action model that keeps all six camera views while enabling efficient inference by discarding the video branch at deployment. It uses future‑video prediction as dense training supervision and an action‑centered causal mask to prevent action tokens from attending to future‑video tokens during joint denoising. A differentiable drivable‑area compliance regularizer penalizes vehicle‑footprint corners near or crossing drivable boundaries, improving safety and boundary awareness. Experiments on NAVSIMv2 and nuScenes show state‑of‑the‑art planning performance with low latency and strong zero‑shot transfer.