WorldAgen is a unified framework that jointly learns world modeling and action prediction using a shared Transformer backbone with two specialized heads. It introduces a Mixed Unidirectional Attention Mask to separate the world model and agent model, and enables Test-Time Training (TTT) by sampling exploratory actions and updating the world model with real state transitions. Experiments on CALVIN and LIBERO show that WorldAgen matches or surpasses state‑of‑the‑art methods, especially when TTT is applied to a few samples.
By Chi Wan, Kangrui Wang, Yuan Si, Pingyue Zhang, Manling Li
arXiv:2610.27280v1 Announce Type: cross
Abstract: Offline goal-conditioned reinforcement learning enables skill acquisition by reaching any goal from any state without dense rewards. Model-based meth...
By Yunzhe Hu, Dong Xu
arXiv:2609.34981v2 Announce Type: replace
Abstract: World action models (WAMs) predict the future alongside actions during training. Due to the heavy computation cost of video denoising, whether the...
By Renping Zhou, Zanlin Ni, Zihao Fan, Guohao Fu, Zeyu Liu, Hao Shi, Jie Zhang, Chi Bene Chen, Yang Yue, Xueyang Fu, Gao Huang
The paper introduces the AGI Maze Prediction Datasets and Benchmark, a lightweight, procedurally generated grid‑world testbed for evaluating predictive models, particularly Transformers, on tasks such as per‑step transition prediction, fixed‑horizon state prediction, and sequential textual‑observation prediction. It compares byte‑level Transformer baselines with two memory‑augmented architectures, showing that a pseudo‑video spatial‑memory Transformer achieves perfect validation accuracy on selected tasks and improves sequential text‑trace prediction, while a generic auxiliary latent‑memory Transformer does not consistently help. The study highlights that structured, task‑aligned working memory can be more effective than merely increasing latent capacity, and positions the benchmark as a compact setting for testing architectures that couple textual interfaces to learned structured state.
By Alexey Potapov
arXiv:2611.05847v1 Announce Type: cross
Abstract: Joint-Embedding Predictive Architectures (JEPAs) support planning by comparing predicted latent states with an encoded goal. For this comparison to g...
By Ruilong Yu, Fei Ye
Offline goal-conditioned reinforcement learning enables skill acquisition by reaching any goal from any state without dense rewards. Model-based methods allow for planning by scaling test-time compute...
arXiv:2610.33941v1 Announce Type: new
Abstract: World action models have demonstrated promising performance in robot policy learning, yet their high inference latency remains a critical bottleneck fo...
By Guile Wu, Shuangzhi Li, Xingxin Chen, Chengjie Huang, Sam Hosseini, Tongtong Cao, Yingxue Zhang, Dongfeng Bai
The paper introduces the Dual-Latent World Model (Dual-WM), which separates local execution and long-range planning into distinct latent spaces and dynamics models. A new learning method, Long-Horizon Representation Learning with Weighted Rollout (LoRe), supervises predictions at both levels using exponential horizon weights. Experiments on five goal-conditioned visual control tasks show that Dual-WM improves success rates over strong baselines, especially at longer horizons.
By Delin Zhao, Zhengrong Yue, Shaobin Zhuang, Junlin He, Xiaoyu Chen, Zikang Wang, Yuxin Liu, Limin Wang, Yali Wang
arXiv:2610.28122v1 Announce Type: cross
Abstract: Cascade world-action models (WAMs) use predicted visual futures to condition robot actions, but inherit costly iterative inference from diffusion-pre...
By Shaoyi Zheng, Zhongtao Jiang, Yue Peng, Wenbo Lu, Shunchi Zhang, Xi Wang, Shengji Wan
SWIM (State Weighting with Inverse-Dynamics Model) learns which state features matter for joint-embedding predictive architectures (JEPAs) by training an inverse-dynamics model on offline trajectories to predict actions from states and future displacements. The resulting feature weights define a weighted state metric that aligns latent and state distances, improving state-supervised training without extra planning-time computation. Experiments on five simulated robotic manipulation tasks show that SWIM consistently outperforms uniform state weighting in planning success.
By Jiaming Hu, Shi Bo, Yan Zheng, Tian Wang, Florian Dubost, Alejandro Mottini, Junze Liu, Arvind Srinivasan, Kai Zhong, Kun Qian, Sharon Gao, Vianne Gao, Houyu Zhang, Qingjun Cui
The paper introduces the AGI Maze Prediction Datasets and Benchmark, a lightweight testbed for evaluating how Transformers and other models learn world dynamics. The benchmark, built from procedurally generated grid worlds, includes per‑step transition prediction, fixed‑horizon state prediction, and sequential textual‑observation prediction, with source‑maze‑disjoint training and validation splits to test transferable action‑conditioned dynamics. Experiments show that a pseudo‑video spatial‑memory Transformer, which initializes and updates a two‑dimensional latent workspace from the input map and action history, achieves perfect validation accuracy on selected tasks and improves sequential text‑trace prediction, outperforming byte‑level and unstructured‑memory baselines and suggesting that structured, task‑aligned working memory is more effective than additional latent capacity alone.
DeepJEPA is a weight‑tied joint‑embedding predictive world model that treats transition depth as an inner test‑time scaling axis, learning when additional recurrent updates are worthwhile for each candidate and rollout step. Unlike traditional planners that uniformly deepen every transition, DeepJEPA concentrates extra computation on decision‑critical events such as contact onset and sustained object interaction, achieving comparable or better performance with only 1.00–1.26 updates per transition across five visual‑control settings. The approach demonstrates that improved planning does not require uniformly better object‑state decodability, but rather targeted internal computation where it can alter the planner’s elite set and action selection.
By Zijian Jin, Yunbei Zhang, Yuanzhe Liu, Ming Liu, Baian Chen, Weirui Ye, Shilong Liu, Marco Pavone