arXiv Computer Vision By Shashank Hegde, Alexander Popov, Elie Aljalbout, Nikolai Smolyanskiy

Diffusion-Steering-Based MPC with a JEPA World Action Model

Read the original on arXiv Computer Vision →

The paper introduces LeWAM, a bidirectional transformer that predicts forward, backward, inverse dynamics, and policy using a decoder‑free JEPA latent trained end‑to‑end. LeWAM’s latent representation aligns better with robot and object states than traditional reconstruction‑based world action models and ignores visual distractors more effectively. In closed‑loop experiments, LeWAM matches the performance of a flow‑matching policy while also providing a world model, and planning in the policy head’s noise space improves MPC performance compared to sampling raw actions.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computer Vision.

arXiv AI
Oct 9

LeWAM: A JEPA World Action Model with Diffusion-Steering-Based MPC

LeWAM is a bidirectional transformer that predicts forward, backward, inverse dynamics and policy, trained end‑to‑end on a decoder‑free JEPA latent. It outperforms a forward‑only JEPA world model by better aligning latent representations with robot and object states while ignoring visual distractors. In closed‑loop tests, LeWAM matches a flow‑matching policy of the same size and improves MPC planning by sampling actions in the policy head’s noise space rather than raw actions.

By Shashank Hegde, Alexander Popov, Elie Aljalbout, Nikolai Smolyanskiy
arXiv AI
Oct 12

LeWAM: End-to-End World and Action Modeling with JEPAs

LeWAM is an end‑to‑end joint‑embedding predictive architecture that learns compact latent world models without pixel reconstruction and directly supports policy execution and planning. It trains jointly on next‑latent‑state prediction and action‑flow matching while imposing an isotropic Gaussian constraint on the latent state. The approach yields significant performance gains, raising visual‑planning success rates from 28.6 % to 89.7 % on contact‑rich tasks, from 10.9 % to 46.2 % on long‑horizon tasks, and speeding up planning by 32.3×, while remaining competitive on behavior‑cloning tasks and providing interpretable control‑relevant representations.

By Minghao Fu, Tavis Siebert, Eryk Halicki, Randall Balestriero
arXiv Machine Learning
Oct 12

SmolWAM: A Latent World-Action Model for Robotic Manipulation on Edge Devices

SmolWAM is a compact latent world‑action model designed for robotic manipulation on edge devices. It is derived from the 2B‑parameter Cosmos Policy by reusing its pretrained trunk and selecting 12 of 28 transformer blocks, while employing single‑frame visual latents and cached text features to produce actions and future visual latents in a single rectified‑flow step. Trained in only 3.6 GPU‑hours, SmolWAM achieves 95.5% success on 400 LIBERO episodes at 57 ms per step, using 2.3 TFLOP per 16‑action chunk and 2.4 GiB GPU memory—an 11× speed‑up, 18.5× fewer FLOPs, and 3.2× less memory than its teacher model, while still forecasting scene motion.

By Timi Owolabi, Hikmah Olawore, Frances Adelakun, Ayotomiwa Oyewumi
arXiv AI
Jun 16

LaWAM: Latent World Action Models for Efficient Dynamics-Aware Robot Policies

arXiv:2606. 15768v1 Announce Type: cross Abstract: Vision-Language-Action models (VLAs) leverage large-scale vision-language pretraining for semantic robot control, but often lack explicit foresight into how robot actions change the scene.

By Jialei Chen, Kai Wang, Kang Chen, Shuaihang Chen, Feng Gao, Wenhao Tang, Zhiyuan Li, Weilin Liu, Zhuyu Yao, Boxun Li, Yuanbo Xu, Chao Yu
arXiv Machine Learning
Jun 26

Fast LeWorldModel

arXiv:2606. 26217v1 Announce Type: new Abstract: Joint-Embedding Predictive Architectures (JEPAs), including recent LeWorldModel (LeWM), have become a promising foundation for reconstruction-free visual world models.

By Yuntian Gao, Xiangyu Xu