The paper investigates whether latent world models truly capture the underlying dynamics of their environments. It introduces the idea of faithful latent world models, defined as an equivariance between the environment’s transition map and the learned latent dynamics, and proposes latent rollout consistency (LRC) as a metric for evaluating action‑conditioned latent predictions. Experiments show that none of the state‑of‑the‑art latent world models studied actually faithfully model the environment’s dynamics, despite solving the tasks they were trained on.
The paper presents a reward‑free continual learning framework for space robots that uses latent‑state world models to adapt to severe hardware degradation. By pre‑training a model‑based agent in diverse simulations, the world model learns to predict reward structure in latent space. During deployment, the observation encoder and reward predictor are frozen while only the transition dynamics are updated via unsupervised rollouts, allowing the policy to adapt using imagined trajectories without new rewards.
By Andrej Orsula, Miguel Olivares-Mendez, Carol Martinez
The paper introduces the notion of faithful latent world models, defined as an equivariance between an environment’s transition map and the learned latent dynamics. It proposes latent rollout consistency (LRC) as a metric for evaluating action‑conditioned latent predictions. Experiments show that current state‑of‑the‑art latent world models fail to faithfully model the environment’s dynamics, despite achieving task performance.
By Rodrigo Gonzalez Laiz, Steffen Schneider
arXiv:2608.23526v1 Announce Type: new
Abstract: World models can predict video without learning dynamics that they reliably preserve. We test whether a frozen DreamerV3 trained only on pendulum video...
By Richard Bao
arXiv:2606. 20104v1 Announce Type: cross Abstract: Perception for action suggests that representations of the world should be shaped not by visual fidelity alone, but by their relevance for actions.
By Petr Ivashkov, Randall Balestriero, Bernhard Sch\"olkopf
arXiv:2611.07245v1 Announce Type: cross
Abstract: We study whether \emph{long-horizon model-based value expansion} can serve as an effective framework in pixel-based offline reinforcement learning (R...
By Kwanyoung Park, Sergey Levine
arXiv:2607. 12474v1 Announce Type: new Abstract: Recent advances in foundation models have transformed AI for Science, enabling remarkably accurate predictive performance across domains ranging from protein folding to weather forecasting.
By Ingmar Posner, Anson Lei, Bernhard Sch\"olkopf
arXiv:2603. 19312v3 Announce Type: replace Abstract: Joint Embedding Predictive Architectures (JEPAs) offer a compelling framework for learning world models in compact latent spaces, yet existing methods remain fragile, relying on complex multi-term losses, exponential moving averages, pre-trained encoders, or auxiliary supervision to avoid representation collapse.
By Lucas Maes, Quentin Le Lidec, Damien Scieur, Yann LeCun, Randall Balestriero
arXiv:2605. 00412v3 Announce Type: replace Abstract: World models have recently re-emerged as a central paradigm for embodied intelligence, robotics, autonomous driving, and model-based reinforcement learning.
By Sen Cui, Jingheng Ma
T2-Z introduces a test‑time optimization framework that augments robotic world models with a learnable environment latent, enabling the model to adapt to unseen environmental conditions without explicit supervision of those factors. During training, the method alternates between optimizing the world‑model dynamics and the latent representation, allowing the latent to capture underlying environmental variation. At test time, a freshly initialized latent is optimized from a few interactions, reducing prediction error from 15.20 to 7.00 and increasing action success from 33.73% to 72.96% across simulated and real‑world tasks.
By Chenyuan Zhou, Yining Hong, Huang Huang, Yunfan Jiang, Wenlong Huang, David Held, Vitor Campagnolo Guizilini, Yue Wang, Li Fei-Fei, Jiajun Wu, Leonidas Guibas, Yejin Choi
WorldTS is a new forecasting framework that models latent dynamics conditioned on multimodal covariates to improve time‑series prediction. It uses a two‑stage training process: first learning latent state dynamics from historical data and covariates, then training a decoder to map predicted latent states back to future observations. Experiments on 21 real‑world datasets demonstrate the effectiveness of this approach.
By Yuhan Zhu, Xiangfei Qiu, Hanyin Cheng, Wangmeng Shen, Chenjuan Guo, Bin Yang, Jilin Hu, Christian S. Jensen
In the physical world we inhabit, space and time are fundamentally continuous. However, existing machine learning paradigms for world modeling are largely confined to discrete-time prediction, thereby exhibiting significant inefficiency in capturing the dynamics of physical world.