arXiv Computer Vision

Diffusion-Steering-Based MPC with a JEPA World Action Model

The paper introduces LeWAM, a bidirectional transformer that predicts forward, backward, inverse dynamics, and policy using a decoder‑free JEPA latent trained end‑to‑end. LeWAM’s latent representation aligns better with robot and object states than traditional reconstruction‑based world action models and ignores visual distractors more effectively. In closed‑loop experiments, LeWAM matches the performance of a flow‑matching policy while also providing a world model, and planning in the policy head’s noise space improves MPC performance compared to sampling raw actions.

arXiv AI
Oct 9

LeWAM: A JEPA World Action Model with Diffusion-Steering-Based MPC

LeWAM is a bidirectional transformer that predicts forward, backward, inverse dynamics and policy, trained end‑to‑end on a decoder‑free JEPA latent. It outperforms a forward‑only JEPA world model by better aligning latent representations with robot and object states while ignoring visual distractors. In closed‑loop tests, LeWAM matches a flow‑matching policy of the same size and improves MPC planning by sampling actions in the policy head’s noise space rather than raw actions.

By Shashank Hegde, Alexander Popov, Elie Aljalbout, Nikolai Smolyanskiy
arXiv AI
Oct 12

LeWAM: End-to-End World and Action Modeling with JEPAs

LeWAM is an end‑to‑end joint‑embedding predictive architecture that learns compact latent world models without pixel reconstruction and directly supports policy execution and planning. It trains jointly on next‑latent‑state prediction and action‑flow matching while imposing an isotropic Gaussian constraint on the latent state. The approach yields significant performance gains, raising visual‑planning success rates from 28.6 % to 89.7 % on contact‑rich tasks, from 10.9 % to 46.2 % on long‑horizon tasks, and speeding up planning by 32.3×, while remaining competitive on behavior‑cloning tasks and providing interpretable control‑relevant representations.

By Minghao Fu, Tavis Siebert, Eryk Halicki, Randall Balestriero
arXiv Machine Learning
Oct 12

SmolWAM: A Latent World-Action Model for Robotic Manipulation on Edge Devices

SmolWAM is a compact latent world‑action model designed for robotic manipulation on edge devices. It is derived from the 2B‑parameter Cosmos Policy by reusing its pretrained trunk and selecting 12 of 28 transformer blocks, while employing single‑frame visual latents and cached text features to produce actions and future visual latents in a single rectified‑flow step. Trained in only 3.6 GPU‑hours, SmolWAM achieves 95.5% success on 400 LIBERO episodes at 57 ms per step, using 2.3 TFLOP per 16‑action chunk and 2.4 GiB GPU memory—an 11× speed‑up, 18.5× fewer FLOPs, and 3.2× less memory than its teacher model, while still forecasting scene motion.

By Timi Owolabi, Hikmah Olawore, Frances Adelakun, Ayotomiwa Oyewumi
arXiv AI
Jun 16

LaWAM: Latent World Action Models for Efficient Dynamics-Aware Robot Policies

arXiv:2606. 15768v1 Announce Type: cross Abstract: Vision-Language-Action models (VLAs) leverage large-scale vision-language pretraining for semantic robot control, but often lack explicit foresight into how robot actions change the scene.

By Jialei Chen, Kai Wang, Kang Chen, Shuaihang Chen, Feng Gao, Wenhao Tang, Zhiyuan Li, Weilin Liu, Zhuyu Yao, Boxun Li, Yuanbo Xu, Chao Yu
arXiv Machine Learning
Jun 26

Fast LeWorldModel

arXiv:2606. 26217v1 Announce Type: new Abstract: Joint-Embedding Predictive Architectures (JEPAs), including recent LeWorldModel (LeWM), have become a promising foundation for reconstruction-free visual world models.

By Yuntian Gao, Xiangyu Xu
arXiv AI
Sep 4

Toward Physically Grounded JEPA World Models for Goal-Conditioned Robotic Planning

The paper presents an end‑to‑end JEPA world model that enhances latent prediction with inverse dynamics and state alignment to improve goal‑conditioned robotic planning. By preventing latent collapse and grounding representations in physical configuration, the model achieves top success rates on tasks such as TwoRoom, PushT, and OGBench‑Cube, outperforming the baseline LeWorldModel. Ablation studies confirm that state alignment consistently boosts planning success over inverse dynamics alone across all four benchmark tasks.

By Muyuan Liu (GENISOM AI, Beijing, China), Yue Huang (GENISOM AI, Beijing, China), Zheng Liang (GENISOM AI, Beijing, China), Xiang Gao (GENISOM AI, Beijing, China)
arXiv Computer Vision
Sep 24

AWM-VLA: AlignedWorld Modeling for Efficient and Explainable Vision-Language-Action Policies

AWM‑VLA introduces a unified framework that embeds aligned world modeling directly into a diffusion‑transformer vision‑language‑action policy. By adding learnable future tokens aligned with vision‑language embeddings of future observations, the policy can anticipate long‑term consequences while generating actions. The method extends this with an object‑centric alignment objective and a principled weighting scheme, achieving up to 21% higher success rates on RoboCasa and humanoid tabletop benchmarks and producing object‑centric rationales preferred by human raters in 83% of cases.

By An Lanji, Dawei Liu, Jin Li, Haoran Xu, Mei Chen, Yu Tian
arXiv AI
Oct 23

Feedback World Model Enables Precise Guidance of Diffusion Policy

The paper introduces a feedback world model that updates predictions online using the robot’s observed next state after each action, thereby closing the loop between prediction and observation. This lightweight feedback state corrects future predictions without extra training or parameter updates, and the authors provide convergence guarantees under bounded dynamics residuals. Experiments on LIBERO‑Plus, Robomimic, and real‑world tasks show significant reductions in prediction error (up to 76.4%) and improved policy performance, even when applied to pretrained foundation models like Fast‑WAM‑Joint.

By Tuo An, Jindou Jia, Gen Li, Jingliang Li, Chuhao Zhou, Pengfei Liu, Bofan Lyu, Jiaqi Bai, Xinying Guo, Geng Li, Jianfei Yang