arXiv AI
Vision-Language Agents Can Learn in Their Own Latent Worlds
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv AI
Aug 28
Decoupling Planning and Control for Instructable Agents
The paper introduces Instruct-to-Act, a system that decouples planning and control by combining a vision‑language model (VLM) planner with a world‑model controller. The VLM generates sparse, high‑level text instructions, while the controller executes them at high frequency, trained via relabeling rollouts with synthetic instructions and joint optimization of behavior cloning, reward, and world‑model objectives. Across seven embodied environments—including multi‑agent settings—this approach outperforms controller‑only and direct VLM action methods, maintains fast control, and allows swapping pretrained VLM planners without fine‑tuning, achieving competitive results with strong baselines on most tasks.
Hugging Face Trending Papers
Aug 27
Decoupling Planning and Control for Instructable Agents
The paper introduces Instruct-to-Act, a system that decouples high‑level planning from low‑latency control by combining a vision‑language model (VLM) planner with a world‑model controller. The VLM generates sparse, high‑level text instructions, while the controller executes them autonomously at high frequency. Experiments across seven embodied environments, including multi‑agent settings, show that this approach outperforms both controller‑only and direct VLM action‑generation methods, maintains fast control, and allows swapping in different pretrained VLM planners without fine‑tuning.
arXiv AI
Jun 30
WoVR: World Models as Reliable Simulators for Post-Training VLA Policies with RL
arXiv:2602. 13977v2 Announce Type: replace-cross Abstract: Reinforcement learning (RL) promises to unlock capabilities beyond imitation learning for Vision--Language--Action (VLA) models, but its requirement for massive real-world interaction prevents direct deployment on physical robots.
arXiv Machine Learning
Oct 21
LaLa-World: Scaling Latent World Models with Natural Language Actions
arXiv:2610.27851v1 Announce Type: cross Abstract: This paper shows that JEPA-style latent world models (LWMs) can learn across diverse domains and data sources by adopting the language-conditioned in...
arXiv AI
Oct 5
Latent Goal Prediction from Language for Model-Based Planning
arXiv:2606.20627v2 Announce Type: replace Abstract: Joint-Embedding Predictive Architectures (JEPAs) enable agents to plan in latent space by imagining the outcomes of candidate actions, yet task spe...
Hugging Face Trending Papers
Oct 22
DRILL: Training World Models to Improve Policies, Not Predict Pixels
The paper introduces DRILL, a bilevel training framework that optimizes world models for policy improvement rather than pixel accuracy. DRILL-VWI reweights prediction errors using policy visitation and model confidence, while DRILL-IMG employs meta‑gradients to directly maximize downstream policy returns. Experiments on manipulation simulators show DRILL-IMG boosts success rates by 13.1 points over WMPO and 7.9 points over RLVR-World, demonstrating that world models can be more effectively trained using policy gradients.