arXiv AI

Where Predictive Supervision Goes Shapes What VLA Policies Learn

arXiv AI
Oct 16

Learning Intrinsic Dynamics in Latent World Models

The paper investigates how encouraging action predictability in latent world models can preserve intrinsic dynamics, which are crucial for planning. By deriving a theoretical lower bound and conducting empirical tests, the authors show that action predictability prevents the encoder from merging distinct action outcomes, even when visual appearance varies. Using a method called Concat IDM, they achieve a substantial increase in planning success on the Push‑T benchmark, from 9.1% to 77.2%, demonstrating the practical benefit of this inductive bias.

By Jiamin Shi, Yuanhong Zhang, Zhiyang Liu, Jiawei Sun, Jingwen Fu
arXiv AI
Jun 16

LaWAM: Latent World Action Models for Efficient Dynamics-Aware Robot Policies

arXiv:2606. 15768v1 Announce Type: cross Abstract: Vision-Language-Action models (VLAs) leverage large-scale vision-language pretraining for semantic robot control, but often lack explicit foresight into how robot actions change the scene.

By Jialei Chen, Kai Wang, Kang Chen, Shuaihang Chen, Feng Gao, Wenhao Tang, Zhiyuan Li, Weilin Liu, Zhuyu Yao, Boxun Li, Yuanbo Xu, Chao Yu
arXiv AI
Oct 31

SaiVLA-JEPA: Evaluating Future Latent Prediction in JEPA-Conditioned VLA Control

SaiVLA-JEPA is a family of vision‑language‑action policies that condition on frozen V‑JEPA features and use lightweight auxiliary heads to predict future latent representations during training. Experiments on 10,030 LIBERO‑Plus episodes show that JEPA conditioning improves success rates to about 82.5% versus 71.6% without JEPA, but adding future supervision does not yield a detectable aggregate gain. Variants that adapt latent context or selected parameters provide mixed results, suggesting that latent‑prediction objectives should be evaluated via closed‑loop rollouts rather than latent loss alone.

By Rufeng Xiao (VLA Research Team, Synthoid.ai), Ruihao Xiao (VLA Research Team, Synthoid.ai), Wei Chen (VLA Research Team, Synthoid.ai), Xinhai Sun (VLA Research Team, Synthoid.ai)
arXiv AI
Sep 17

PACT-WAM: Predicting Actions and Visual Foresight with Compact Temporal Encoding for Robot Manipulation

PACT‑WAM is a world‑action model that simultaneously predicts a 16‑step action trajectory and its corresponding visual forecast for robot manipulation. It uses a hierarchical history encoder that compresses past observations into fewer tokens, reducing processing cost by 75% compared to dense encoding. The model’s shared flow module updates action and visual states jointly, and a TiTok‑VAE decoder reconstructs multi‑view future images, which are then used by a vision‑language component (Proposal Review) to improve execution‑prefix selection and proposal rejection, boosting success rates on several benchmarks.

By Yushan Liu, Jingjing Fan, Shoujie Li, Yifan Xie, Xiao-Ping Zhang, Wenbo Ding
arXiv AI
Jul 21

Anticipate Before Acting: Future-State-Conditioned Vision-Language Navigation

arXiv:2607. 18042v1 Announce Type: cross Abstract: End-to-end vision-language navigation (VLN) with causal vision-language models can map instructions and egocentric observations directly to actions, but standard behavior cloning supervises only the next action and does not explicitly train the policy state to be predictive of future visual outcomes.

By Lingfeng Zhang, Zhanguang Zhang, Liheng Ma, Tongtong Cao, Yingxue Zhang
arXiv AI
Oct 28

Vision-Language Agents Can Learn in Their Own Latent Worlds

The paper introduces Dwell, a model‑based reinforcement learning framework that treats a pretrained vision‑language model (VLM) as a recurrent latent transition operator. Dwell enables the VLM to predict, adjust, and act within its own latent space, allowing the agent to learn from real interactions and latent imagination without reconstructing full futures. Experiments across five interactive visual environments show that Dwell outperforms other trainable methods while significantly reducing computational demands for imagination.

By Haoxi Li, Sikai Bai, Jianfei Ma, Qinglin Hou, Xiaocheng Lu, Tao Han, Jie Zhang, Song Guo
arXiv Computer Vision
Sep 21

MT-WAM: Reorienting the One-Pass Predictive Representation Toward Action Generation

MT‑WAM enhances the Fast‑WAM framework by adding complementary supervision for future 2‑D point trajectories and visual features while keeping the original training objectives. A lightweight dual‑stream branch and structured attention mask isolate motion‑specific processing, and motion‑stream tokens provide additional dynamics cues to the action expert. During inference, MT‑WAM skips future‑video prediction, using cached video and motion information to achieve higher success rates on LIBERO, LIBERO‑Plus, RoboTwin 2.0 Clean2Rand, and several real‑world tasks.

By Yiguang Yang, Jiankun Peng, Xiaoming Wang, Yiran Zhang, Zhibo Fang