The paper introduces Critic-Guided Synthetic Experience (CGSE), a method that uses a frozen latent world model to generate new transitions offline. CGSE samples candidate actions from an offline policy, refines them to maximize a first‑order approximation of the Q‑function within a trust region, and then uses the critic to select the best action for each imagined transition. The resulting synthetic transitions are added to the replay buffer, improving an offline policy’s success rate by 7.1% on three OGBench manipulation tasks.
arXiv:2607. 17973v1 Announce Type: new Abstract: Latent world models have emerged as a powerful planning paradigm by learning action-conditioned predictive dynamics and using them as internal simulators to imagine and evaluate candidate action sequences.
The paper introduces QWM, a framework that integrates world models with standard Q‑learning to perform test‑time search over imagined trajectories. By training the policy and value function solely on real transitions, QWM avoids compounding model bias while still benefiting from predictive search. Experiments on the Robomimic and LIBERO manipulation benchmarks show that QWM outperforms strong prior state‑of‑the‑art methods in both sample efficiency and performance.
Offline goal-conditioned reinforcement learning enables skill acquisition by reaching any goal from any state without dense rewards. Model-based methods allow for planning by scaling test-time compute...
arXiv:2610.27280v1 Announce Type: cross Abstract: Offline goal-conditioned reinforcement learning enables skill acquisition by reaching any goal from any state without dense rewards. Model-based meth...
MAC‑v2 is a new framework for long‑horizon model‑based value expansion from pixel observations in offline reinforcement learning. It trains a reconstruction‑free latent world model that learns representations through prediction and control objectives, avoiding the need to encode control‑irrelevant visual details. By using action chunking, MAC‑v2 reduces recursive model predictions, mitigating error accumulation and achieving strong performance on visual benchmarks such as OGBench, MMBench2, Robomimic, and LIBERO‑10.
The paper introduces DRILL, a bilevel training framework that optimizes world models for policy improvement rather than pixel accuracy. By reweighting prediction errors with policy visitation and confidence (DRILL‑VWI) or using a full meta‑gradient approach (DRILL‑IMG), the authors demonstrate significant gains in downstream policy success rates across multiple manipulation simulators. The work shows that world models can be more effectively trained using the policy gradients they generate, rather than relying solely on pixel‑level prediction metrics.
The paper introduces DRILL, a bilevel training framework that optimizes world models for policy improvement rather than pixel accuracy. DRILL-VWI reweights prediction errors using policy visitation and model confidence, while DRILL-IMG employs meta‑gradients to directly maximize downstream policy returns. Experiments on manipulation simulators show DRILL-IMG boosts success rates by 13.1 points over WMPO and 7.9 points over RLVR-World, demonstrating that world models can be more effectively trained using policy gradients.
FIRM-WM is a compact pixel world model that separates a goal‑comparable configuration from a 128‑dimensional dynamic fiber, enabling reward‑free visual planning from offline videos. It addresses two key mismatches: aligning planning states with goal images and reconciling factual trajectories with interventional sampling. In experiments, FIRM‑WM achieves high success rates on TwoRoom, Reacher, and OGBench‑Cube while using fewer parameters and faster planning times than prior models.
The paper introduces Dwell, a model‑based reinforcement learning framework that treats a pretrained vision‑language model (VLM) as a recurrent latent transition operator. Dwell enables the VLM to predict, adjust, and act within its own latent space, allowing the agent to learn from real interactions and latent imagination without reconstructing full futures. Experiments across five interactive visual environments show that Dwell outperforms other trainable methods while significantly reducing computational demands for imagination.
arXiv:2606. 07974v1 Announce Type: cross Abstract: A learned world model provides a powerful physical intuition for evaluating future states.
PAVXploreRL introduces a reinforcement learning framework that builds on a pretrained latent world model to explicitly optimize Physical Plausibility, Action Adherence, and Visual Fidelity (PAV) objectives. By combining in‑distribution expert trajectories with noise‑driven out‑of‑distribution action exploration, the method avoids reliance on paired video supervision and improves generalization. Experiments demonstrate a 5.6% average performance gain over pretrained baselines and more reliable policy evaluation with reduced overestimation bias.
Continually improving agents require dynamic interaction feedback beyond static supervision, yet direct real-environment interaction is costly, slow, unsafe, and hard to parallelize. World modeling offers a natural intermediate proxy that allows agents to query lower-cost, more controllable feedback before committing to real actions.