arXiv Machine Learning
Jun 18
Hierarchical Planning with Latent World Models
arXiv:2604. 03208v2 Announce Type: replace Abstract: World models are a promising path to zero-shot embodied control through planning.
Hierarchical World Models with Implicit Dynamics
Hierarchical World Models with Implicit Dynamics (ID‑WM) introduces a two‑level approach that separates state‑space reachability from local control. A high‑level model samples feasible transitions across the state manifold, while a low‑level world model acts as an inverse dynamics solver to translate these transitions into precise actions. This design enables planners to identify reachable subgoals without being constrained by low‑level execution details, producing novel long‑horizon plans that outperform state‑of‑the‑art policy and diffusion‑based planners by 38.5% and single‑level MPC by 135% in complex, unseen configurations.
Related stories
arXiv AI
Jun 9
Unifying Object-Centric World Models and Diffusion Policy: A Hierarchical Framework for Multi-Stage Robotic Tasks
arXiv:2606. 08775v1 Announce Type: cross Abstract: Visual world models have shown great potential in learning complex system dynamics.
arXiv Machine Learning
Jun 8
Latent Geometry Beyond Search: Amortizing Planning in World Models
arXiv:2605. 08732v2 Announce Type: replace-cross Abstract: Modern vision-based world models can represent observations as compact yet expressive latent manifolds, but fast goal-oriented planning in these spaces remains challenging.
arXiv AI
Sep 15
LePlanner: An Iterative Amortized Controller For World Models
arXiv:2609.13845v1 Announce Type: cross Abstract: World models trained with joint-embedding predictive architectures learn compact, structured latent representations from physical interaction, yet pl...
arXiv AI
Jul 22
Lifting Embodied World Models for Planning and Control
arXiv:2604. 26182v2 Announce Type: replace-cross Abstract: World models of embodied agents predict future observations conditioned on an action taken by the agent.
arXiv AI
Sep 3
What Drives Success in Physical Planning with Joint-Embedding Predictive World Models?
The paper investigates Joint-Embedding Predictive World Models (JEPA-WMs), a class of methods that perform planning in a learned representation space rather than raw input space. It systematically studies how model architecture, training objectives, and planning algorithms influence success across simulated and real‑world robotic tasks, and proposes a JEPA-WM variant that surpasses established baselines in navigation and manipulation. The authors provide code, data, and checkpoints for reproducibility.
arXiv Machine Learning
Aug 11
LUCID: Latent-Skill Unified Control via Imagined Dynamics for Long-Horizon Humanoid Loco-Manipulation
arXiv:2608. 07746v1 Announce Type: new Abstract: Long-horizon humanoid loco-manipulation requires composing versatile whole-body skills and reliable high-level decision making.
arXiv AI
Jul 7
Cortex: A Bidirectionally Aligned Embodied Agent Framework for Long-horizon Manipulation
arXiv:2607. 05377v1 Announce Type: cross Abstract: While recent Vision-Language-Action (VLA) models show promise toward generalist manipulation policies, they struggle with long-horizon tasks due to their Markovian nature-relying solely on current observations.
arXiv AI
Aug 28
Decoupling Planning and Control for Instructable Agents
The paper introduces Instruct-to-Act, a system that decouples planning and control by combining a vision‑language model (VLM) planner with a world‑model controller. The VLM generates sparse, high‑level text instructions, while the controller executes them at high frequency, trained via relabeling rollouts with synthetic instructions and joint optimization of behavior cloning, reward, and world‑model objectives. Across seven embodied environments—including multi‑agent settings—this approach outperforms controller‑only and direct VLM action methods, maintains fast control, and allows swapping pretrained VLM planners without fine‑tuning, achieving competitive results with strong baselines on most tasks.
arXiv AI
Oct 21
Plausible Future, Wrong Choice: Steering World-Action Models toward Intended Future
World-Action Models (WAMs) jointly predict future world states and robot actions for diverse manipulation tasks, but they can suffer from Stage Confusion—generating plausible yet misdirected rollouts that drift to the wrong task stage. TaskWAM introduces a closed-loop, stage-aware framework using Mode Tokens and State Tokens to represent fine-grained task requirements and execution progress, steering generation toward the intended future. The authors also present StageBench, a multi-stage manipulation benchmark, and show through extensive simulation and real-world experiments that TaskWAM reduces confusion, matches state-of-the-art performance, and transfers effectively across various robot embodiments.
arXiv Computer Vision
Sep 25
Representation World Model: Learning States, Transition and Executable Plans in Representation
The Representation World Model (RWM) learns states, transitions, and executable plans directly within a representation space, bypassing traditional explicit dynamics models and action-space search. It uses inverse-dynamics supervision along latent paths to shape the representation geometry, enabling direct planning by constructing a latent path between current and goal states and recovering actions via inverse dynamics. Experiments on continuous-control benchmarks and robotic manipulation tasks demonstrate RWM’s effectiveness and potential for complex embodied control.
arXiv AI
Sep 30
Beyond a single latent space: a dual-latent world model for long-horizon planning
The paper introduces the Dual-Latent World Model (Dual-WM), which separates local execution and long-range planning into distinct latent spaces and dynamics models. A new learning method, Long-Horizon Representation Learning with Weighted Rollout (LoRe), supervises predictions at both levels using exponential horizon weights. Experiments on five goal-conditioned visual control tasks show that Dual-WM improves success rates over strong baselines, especially at longer horizons.