arXiv AI

Grounded World Model: Latent Planning with Language Goals

arXiv Machine Learning
Sep 22

FIRM-WM: State-factorized factual-interventional recurrent modeling for reward-free visual planning

FIRM-WM is a compact pixel world model that separates a goal‑comparable configuration from a 128‑dimensional dynamic fiber, enabling reward‑free visual planning from offline videos. It addresses two key mismatches: aligning planning states with goal images and reconciling factual trajectories with interventional sampling. In experiments, FIRM‑WM achieves high success rates on TwoRoom, Reacher, and OGBench‑Cube while using fewer parameters and faster planning times than prior models.

By Yilun Wu, Yunjian Zhang, Aobo Li, Mujiangshan Wang, Haitao Wu, Aqiang Zhang
arXiv Computer Vision
Oct 9

WorldGuide: Goal-Directed Video World Model for Procedural Task Execution

WorldGuide is a closed‑loop video world model that, given an initial image and a task goal, predicts an atomic action, generates the corresponding video clip, and uses that clip to decide the next action or terminate the task. It trains a Planner to predict the next action or completion from visual progress and an Executor to realize those actions, while a hierarchical visual memory tracks state over long horizons. The authors introduce WorldGuide‑Bench, a dataset of ~59K step‑annotated videos across 245 tasks, and show that WorldGuide outperforms the recent MiniMax‑H3 model on both WorldGuide‑Bench and VideoCraft‑Bench, highlighting the benefit of coupling planning with learned execution for goal‑directed procedural video generation.

By Ankan Deria, Komal Kumar, Hisham Cholakkal, Fahad Shahbaz Khan, Salman Khan
arXiv AI
Oct 21

UniWALM: Unified World Action Language Model for Embodied Robot Control

UniWALM is a unified world-action-language model that jointly learns video, action, and language in a single shared representation, using a mixture-of-transformers backbone with diffusion heads that co-denoise across modalities. By generating language as part of the same pass as action, the model closes a language‑as‑memory loop without an external planner, and introduces Flow Matching with Correction to maintain reliable action streams. Experiments show UniWALM outperforms baselines on compositional pick‑and‑place tasks and memory‑dependent benchmarks, and dramatically improves success on a real‑world Find‑the‑Item task when using generated textual memory.

By Rahul Sajnani, Diego Romeres, Mingyuan Zhang, Kai Gao, Carl Qi, Baichuan Huang, Srinath Sridhar, Dylan Randle, Lillian J. Ratliff, Jose Barreiros, Manikantan Nambi, Jeroen Vanbaar
arXiv Computer Vision
Sep 30

EVO-WAM: Evolving World Action Models through Video-Action Verification

arXiv:2609.38057v1 Announce Type: new Abstract: Improving robot policies on new tasks without collecting additional expert demonstrations remains a central challenge in robot learning. World action m...

By Shiyang Zhou, Xionghao Wu, Wenbo Li, Shenghe Zheng, Jiyao Zhang, Songsong Yu, Yijun Yang, Jianhui Liu, Haoze Sun, Senqiao Yang, Li Jiang, Jingyong Su, Haoyang Huang, Zhuotao Tian
arXiv AI
Sep 18

JEPA-WAM: Connecting Generated Visual Instructions to World Action Models through JEPA Latent Representations

JEPA-WAM enhances World Action Models (WAMs) by pairing text instructions with stochastically generated visual cues, using a text-to-image generator and a frozen V‑JEPA encoder to create dense goal representations. These representations are compressed into goal tokens that condition both video and action experts via cross‑attention, enabling the model to better ground instructions. On a new real‑robot benchmark, JEPA‑WAM attains 87.3%, 74.5%, and 80.9% success rates across in‑distribution, out‑of‑distribution scenes, and out‑of‑distribution instructions, outperforming prior methods by significant margins.

By Tianbin Liu, Jian Zhu, Taiyi Su, Jianjun Zhang, Chong Ma, Zitai Huang, Yi Xu
arXiv Machine Learning
1d ago

Learning Latent Goal Costs for Visual World-Model Planning with an Expanding Curriculum

The paper introduces a method to refine latent goal costs for image‑goal planning by adding a small residual adapter to frozen I‑JEPA embeddings. The adapter learns from successful and failed rollouts without task‑specific labels, improving planner performance on DotWall and PointMaze benchmarks. Experiments show success rates rising from 54% to 98% in DotWall and from 82% to 96% in PointMaze, and better alignment with shortest‑path ordering on unseen layouts.

By Konstantinos Mylonas, Thrasyvoulos Spyropoulos