The paper examines why latent action models (LAMs) struggle when training on unlabeled videos that contain exogenous state such as background clutter. By extending a linear LAM framework to explicitly model exogenous state, the authors analytically show that the standard reconstruction objective causes latent actions to encode future exogenous information, and that focusing the representation on endogenous components mitigates this interference. They also demonstrate that auxiliary objectives like action-supervision encourage consistency across exogenous states, and validate these insights with experiments on both linear and nonlinear LAMs.
arXiv:2609.15189v1 Announce Type: new Abstract: Latent action models (LAMs) learn action representations from unlabeled videos by inferring latent actions from visual transitions and reconstructing f...
arXiv:2606. 30544v1 Announce Type: new Abstract: Latent Action Models (LAMs) learn action-like proxies from observation transitions.
arXiv:2606. 19408v1 Announce Type: new Abstract: Latent actions provide a compact interface between action-free video and downstream decision-making, yet existing Latent Action Models (LAMs) force every transition through a fixed-capacity bottleneck.
Contrastive World Models propose a new method for learning latent dynamics without pixel reconstruction. By replacing observation reconstruction with a Deep InfoMax-like objective that maximizes mutual information between state-action sequences and local patch features of future observations, the approach encourages state representations to retain predictive information while ignoring visually irrelevant details. Experiments show that this method matches existing baselines in simple settings and significantly outperforms them when distractors or natural video backgrounds are present, while also training more efficiently by eliminating the pixel decoder.
arXiv:2603.02935v2 Announce Type: replace Abstract: Offline meta-reinforcement learning seeks to learn a policy that generalizes to new related tasks online. Context-based methods infer a task repres...
arXiv:2606. 31232v1 Announce Type: new Abstract: Learning visual world models for planning requires compact latent dynamics that remain sensitive to actions, yet reconstruction-free joint-embedding objectives can collapse to action-insensitive representations.
The paper introduces Action Forcing, a method that transforms ordinary unlabeled video into action‑supervised training data by extracting egomotion bases through principal component analysis of pixel displacements. This approach yields grounded throttle–yaw control signals without requiring instrumented platforms or manual annotation, and it trains a high‑capacity video model while preventing pixel‑level overfitting via an online latent critic. The authors also critique standard video generation metrics and propose a reference‑free evaluation that measures controllability, plausibility, conjuring, and geometric integrity, showing that their model can reverse, scale, and compose actions despite limited reverse‑action data.
arXiv:2607. 27138v1 Announce Type: cross Abstract: Vision-language-action (VLA) models remain constrained by scarce action-labeled robot data, whereas action-free videos offer abundant observations of physical change.
arXiv:2611.06901v1 Announce Type: new Abstract: JEPA-style world models learn action-conditioned latent dynamics without reconstructing observations, but distributional regularization alone does not...
JEPA-style world models learn action-conditioned latent dynamics without reconstructing observations, but distributional regularization alone does not ensure that their representations preserve inform...
arXiv:2605.07079v2 Announce Type: replace-cross Abstract: World models predict future transitions from observations and actions. Existing works predominantly focus on image generation only. Visual fe...
arXiv:2608. 19613v1 Announce Type: cross Abstract: Latent Action Models (LAMs) have emerged as a promising paradigm for enabling robot learning to leverage large-scale unlabeled videos through latent actions that serve as compact surrogates for physical actions.