arXiv AI

SALA: Semantically Aligned Latent Actions Enable Dual-Axis Scaling of Embodied Foundation Models

SALA introduces a semantically aligned latent action framework that bridges human and robot video data without requiring paired examples. By leveraging language descriptions to estimate semantic similarities, SALA guides human latent actions toward related robot latents while grounding the latent space with robot action reconstruction. This shared latent action space enables dual‑axis scaling: larger general human pre‑training reduces robot action prediction error, and task‑specific human post‑training boosts real‑world robot success.

Hugging Face Trending Papers
4d ago

SALA: Semantically Aligned Latent Actions Enable Dual-Axis Scaling of Embodied Foundation Models

SALA introduces a method for aligning latent actions across human and robot videos by leveraging language descriptions to create semantic similarities, guiding human latent actions toward related robot latents while grounding the space with robot action reconstruction. This approach avoids the need for paired data and improves training efficiency by preventing visual clustering of latent actions. The shared latent action space enables dual‑axis scaling: larger general human video pre‑training reduces robot action prediction error, and task‑specific human video post‑training boosts real‑world robot task success.

arXiv AI
Oct 12

LAPT-Scale: Scaling Embodied Policies with Latent Action Pre-Training on Human Videos

LAPT-Scale introduces a latent action pre‑training framework that leverages large‑scale egocentric human videos to train embodied robotic policies. The method comprises LAPT‑Scale‑LAM, which learns continuous latent actions by modeling visual transitions in human videos, and LAPT‑Scale‑Policy, which aligns these latent actions with robot embodiment and fine‑tunes for specific tasks. Using 23.3K hours of video, the approach improves real‑robot success rates from 63.3% to 85.0% and generalizes to unseen visual conditions.

By Zhiyuan Xiang, Peidong Liu, Mingyang Li, Jiawei Li, Wenhow Li, Wenbo Ding, Xiang Deng
arXiv AI
Jun 2

From Human Videos to Robot Manipulation: A Survey on Scalable Vision-Language-Action Learning with Human-Centric Data

arXiv:2606. 00054v1 Announce Type: cross Abstract: Recent progress in generalizable embodied control has been driven by large-scale pretraining of Vision-Language-Action (VLA) models.

By Zhiyuan Feng, Qixiu Li, Huizhi Liang, Rushuai Yang, Yichao Shen, Zhiying Du, Zhaowei Zhang, Yu Deng, Li Zhao, Hao Zhao, Zongqing Lu, Oier Mees, Marc Pollefeys, Jiaolong Yang, Baining Guo
arXiv Computer Vision
Oct 5

Native Action-Prior Learning from Videos for World Action Models

The paper introduces NAVA-WAM, a method that learns action priors directly from observation-only videos to pretrain action policies for robots. It uses a two-stage training process: first, it pretrains on videos with future‑video flow‑matching supervision to learn action‑relevant priors, and second, it fine‑tunes with action‑labeled demonstrations for robot control. Experiments show that NAVA‑WAM outperforms previous methods in both in‑distribution and out‑of‑distribution scenarios, achieving strong action‑label efficiency and real‑robot generalization.

By Zhaochong An, Fei Zhang, Menglin Jia, Duncan Frost, Zijian Zhou, Yikai Wang, Xudong Wang, Aditya Patel, Belinda Zeng, Tao Xiang, Serge Belongie, Amir Bar, Sen He
arXiv AI
Sep 18

Improving Cross-embodiment Transfer in Latent Action Models with Action-Similarity Supervision

The paper proposes using action‑similarity supervision to improve cross‑embodiment transfer in latent action models (LAMs). By training the similarity between latent actions to match the similarity of ground‑truth robot action sequences—without predicting the actions themselves—the authors reduce sensitivity to background noise and embodiment differences. Experiments on RoboTwin 2.0 show that this approach more than doubles cross‑embodiment success compared to predicting ground‑truth actions, especially when similarities are computed on end‑effector motion and compared across robots.

By Maxime Alvarez, Renzo Caballero, Tatsuya Matsushima, Yusuke Iwasawa, Yutaka Matsuo
Hugging Face Trending Papers
Jun 18

HumanScale: Egocentric Human Video Can Outperform Real-Robot Data for Embodied Pretraining

Embodied foundation models are expected to benefit from data scaling like large language models, but face a much tighter data bottleneck. Teleoperated real-robot trajectories remain the dominant pretraining source due to their precise action supervision and embodiment alignment, yet their scalability is limited by high collection cost, acquisition difficulty, and low behavioral and environmental diversity.

arXiv Computer Vision
Aug 27

Zero-WAM: In-Context World-Action Modeling from Human Videos for Open-Ended Task Generalization

Zero-WAM introduces a causal video-action model that enables robots to perform unseen manipulation tasks by following in-context human video guidance. The authors create HumanGen, a dataset of 74.2K human-robot ICL pairs across 8.6K tasks, and propose an in-context future chunk prediction objective to prevent shortcut learning. In simulation, Zero-WAM attains a 47.0% success rate on seven unseen tasks, outperforming the best video-action baseline by 29.5 percentage points, and demonstrates real‑world generalization to complex, long‑horizon, and fine‑grained tasks.

By Jiaming Zhou, Qihang Zhang, Gangwei Xu, Cunxin Fan, Yujie Zhao, Ruilin Wang, Yiming Luo, Shuai Yang, Xing Zhu, Yujun Shen, Junwei Liang, Yinghao Xu
arXiv Computer Vision
6d ago

OnlyAct: Extracting Action Representations from Foundation Models for World Models

OnlyAct proposes extracting action representations from a pretrained foundation model (V-JEPA 2) instead of learning actions from scratch. It uses a label‑free contrastive adaptation framework to isolate motion while discarding scene information, avoiding reconstruction bottlenecks. The method is evaluated on diverse benchmarks, showing more expressive action representations and better transfer of actions to new scenes.

By Runjia Li, Minghao Chen, Junyu Xie, Philip Torr, Andrea Vedaldi, Tomas Jakab