arXiv Computer Vision

Reconstructing Is Not Acting: Action-Centric Latent Dynamics Modeling

arXiv Computer Vision
Sep 25

RotVLA: Rotational Latent Action for Vision-Language-Action Model

RotVLA introduces a Vision‑Language‑Action framework that replaces discrete latent action encoding with a continuous rotational latent action representation on the group SO(n). This design provides continuity, compositionality, and structured geometry that better capture real‑world action dynamics, and a triplet frame learning scheme enforces meaningful temporal dynamics while preventing degeneration. Trained with 1.7 B parameters on large cross‑embodiment datasets, RotVLA achieves state‑of‑the‑art performance on LIBERO and RoboTwin2.0 benchmarks and shows strong real‑world manipulation results.

By Qiwei Li, Xicheng Gong, Xinghang Li, Peiyan Li, Quanyun Zhou, Hangjun Ye, Jiahuan Zhou, Yadong Mu
arXiv AI
Oct 16

Learning Intrinsic Dynamics in Latent World Models

The paper investigates how encouraging action predictability in latent world models can preserve intrinsic dynamics, which are crucial for planning. By deriving a theoretical lower bound and conducting empirical tests, the authors show that action predictability prevents the encoder from merging distinct action outcomes, even when visual appearance varies. Using a method called Concat IDM, they achieve a substantial increase in planning success on the Push‑T benchmark, from 9.1% to 77.2%, demonstrating the practical benefit of this inductive bias.

By Jiamin Shi, Yuanhong Zhang, Zhiyang Liu, Jiawei Sun, Jingwen Fu
arXiv Computer Vision
Oct 5

Native Action-Prior Learning from Videos for World Action Models

The paper introduces NAVA-WAM, a method that learns action priors directly from observation-only videos to pretrain action policies for robots. It uses a two-stage training process: first, it pretrains on videos with future‑video flow‑matching supervision to learn action‑relevant priors, and second, it fine‑tunes with action‑labeled demonstrations for robot control. Experiments show that NAVA‑WAM outperforms previous methods in both in‑distribution and out‑of‑distribution scenarios, achieving strong action‑label efficiency and real‑robot generalization.

By Zhaochong An, Fei Zhang, Menglin Jia, Duncan Frost, Zijian Zhou, Yikai Wang, Xudong Wang, Aditya Patel, Belinda Zeng, Tao Xiang, Serge Belongie, Amir Bar, Sen He
arXiv AI
Jul 1

Delta-JEPA: Learning Action-Sensitive World Models via Latent Difference Decoding

arXiv:2606. 31232v1 Announce Type: new Abstract: Learning visual world models for planning requires compact latent dynamics that remain sensitive to actions, yet reconstruction-free joint-embedding objectives can collapse to action-insensitive representations.

By Zhenghao Zhang, Yuanxiang Wang, Zhenyu Guan, Yujia Yang, Bingkang Shi, Tianyu Zong, Hongzhu Yi, Guoqing Chao, Xingchen Chen, Tiankun Yang, Chenxi Bao, Tao Yu, Jingjing Zhou, Jungang Xu
arXiv Machine Learning
1d ago

Beyond Prediction Gain: LASR as a Metric for Action-Grounded Latents in World Models

The paper introduces the Latent Action Support Ratio (LASR), a dimensionless metric designed to evaluate whether latent representations in world models encode action-related information beyond simple predictive shortcuts. LASR offers two summaries—GeoLASR for typical support and WorstLASR for the weakest case—using offline evaluation on held-out data with recorded actions and state changes as targets. Experiments on robot manipulation data and a multi-task panel show that LASR can detect genuine action information, reject non-action shortcuts, and facilitate graded comparisons among imperfect latent representations, also aiding in training and auditing a lightweight residual adapter.

By Yu Zhou, Changsong Lei, Yixiang Wei, Kairui Wen, Huankun Sheng, Ming Li, Yeying Fan, Yu-Hui Wen, Yong-Jin Liu
arXiv Computer Vision
6d ago

OnlyAct: Extracting Action Representations from Foundation Models for World Models

OnlyAct proposes extracting action representations from a pretrained foundation model (V-JEPA 2) instead of learning actions from scratch. It uses a label‑free contrastive adaptation framework to isolate motion while discarding scene information, avoiding reconstruction bottlenecks. The method is evaluated on diverse benchmarks, showing more expressive action representations and better transfer of actions to new scenes.

By Runjia Li, Minghao Chen, Junyu Xie, Philip Torr, Andrea Vedaldi, Tomas Jakab