arXiv AI
Oct 28

TAPESTRY: A Single Backbone for Video Generation and Robot Control

TAPESTRY repurposes a pretrained video diffusion transformer into a robotic policy by treating robot actions as single-patch tokens in the model’s latent space, eliminating the need for video generation at inference. The lightweight linear projection maps each action timestep to the VAE patch dimensions, and the shared backbone is trained with three diffusion objectives—video diffusion, joint video+action diffusion, and action-only diffusion—to learn dynamics priors and action denoising. At test time, the model produces smooth action chunks in a single DDIM step, achieving strong performance on simulation and real‑robot benchmarks with significantly lower inference cost than comparable joint video–action policies.

By Jeremy A. Collins, Seungjae Lee, Rachanon Wachakorn, Krishnan Srinivasan, Animesh Garg, Vitor Campagnolo Guizilini, Paarth Shah
arXiv Computer Vision
Sep 21

MT-WAM: Reorienting the One-Pass Predictive Representation Toward Action Generation

MT‑WAM enhances the Fast‑WAM framework by adding complementary supervision for future 2‑D point trajectories and visual features while keeping the original training objectives. A lightweight dual‑stream branch and structured attention mask isolate motion‑specific processing, and motion‑stream tokens provide additional dynamics cues to the action expert. During inference, MT‑WAM skips future‑video prediction, using cached video and motion information to achieve higher success rates on LIBERO, LIBERO‑Plus, RoboTwin 2.0 Clean2Rand, and several real‑world tasks.

By Yiguang Yang, Jiankun Peng, Xiaoming Wang, Yiran Zhang, Zhibo Fang
Hugging Face Trending Papers
Oct 9

What Visual Representations Do World Action Models Need?

The paper investigates how different visual representations and integration mechanisms affect the performance of World Action Models (WAMs). It finds that strong general‑purpose visual representations yield the largest robustness gains, that channel‑wise joint prediction offers the best success rate, and that additional sources should only be added if they contribute task‑relevant information beyond existing representations. The authors propose a Representation Recipe (R$^2$‑WAM) that achieves state‑of‑the‑art success rates on RoboCasa GR1 Tabletop and RoboTwin 2.0.

arXiv Computer Vision
Oct 12

What Visual Representations Do World Action Models Need?

The paper investigates how different visual representations and integration mechanisms affect the performance of World Action Models (WAMs). It finds that strong general‑purpose visual representations yield the largest robustness gains, that channel‑wise joint prediction is the most effective integration strategy, and that additional sources should be chosen based on the unique task‑relevant information they provide. Using these insights, the authors propose R$^2$‑WAM, which achieves state‑of‑the‑art success rates on RoboCasa GR1 Tabletop and RoboTwin 2.0.

By Kai Liu, Jiangran Lyu, Tingyu Deng, Yixin Zheng, Jingtian Wan, Wenbo Cui, Mi Yan, Yizhou Zhou, Haoran Li, Dongbin Zhao, Zhizheng Zhang, He Wang