← Back to all news
arXiv Computer Vision By Jung Min Lee, Taehyun Cho, Li Zhao, Jungwoo Lee

Why Latent Actions Fail, and How to Prevent It

Read the original on arXiv Computer Vision →

The paper examines why latent action models (LAMs) struggle when training on unlabeled videos that contain exogenous state such as background clutter. By extending a linear LAM framework to explicitly model exogenous state, the authors analytically show that the standard reconstruction objective causes latent actions to encode future exogenous information, and that focusing the representation on endogenous components mitigates this interference. They also demonstrate that auxiliary objectives like action-supervision encourage consistency across exogenous states, and validate these insights with experiments on both linear and nonlinear LAMs.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computer Vision.

Related stories

arXiv Machine Learning
Sep 22

Contrastive World Models

Contrastive World Models propose a new method for learning latent dynamics without pixel reconstruction. By replacing observation reconstruction with a Deep InfoMax-like objective that maximizes mutual information between state-action sequences and local patch features of future observations, the approach encourages state representations to retain predictive information while ignoring visually irrelevant details. Experiments show that this method matches existing baselines in simple settings and significantly outperforms them when distractors or natural video backgrounds are present, while also training more efficiently by eliminating the pixel decoder.

By Bonnie Li
agentsreinforcement-learning
More like this →
arXiv AI
Jul 1

Delta-JEPA: Learning Action-Sensitive World Models via Latent Difference Decoding

arXiv:2606. 31232v1 Announce Type: new Abstract: Learning visual world models for planning requires compact latent dynamics that remain sensitive to actions, yet reconstruction-free joint-embedding objectives can collapse to action-insensitive representations.

By Zhenghao Zhang, Yuanxiang Wang, Zhenyu Guan, Yujia Yang, Bingkang Shi, Tianyu Zong, Hongzhu Yi, Guoqing Chao, Xingchen Chen, Tiankun Yang, Chenxi Bao, Tao Yu, Jingjing Zhou, Jungang Xu
rag
More like this →