arXiv AI October 9, 2026 By Shuang Luo, Yilun Kong, Yunpeng Qing, Yihang Jiao, Zhi Hou, Shunyu Liu, Xiaogang Wang, Dacheng Tao

Rewiring Semantics, Dynamics, and Control: A Simple yet Effective Action-Centric Tri-Stream Transformer

Read the original on arXiv AI →

The paper introduces ACT^3, an Action-Centric Tri-Stream Transformer that merges semantic insights from Vision‑Language Models with dynamics predictions from World Models to generate control actions for robotic manipulation. By allowing a dedicated action expert to attend to both VLM and WM representations while keeping each backbone’s forward pass independent, ACT^3 preserves the distinct roles of context streams. Experiments on simulated and real‑world benchmarks demonstrate that ACT^3 outperforms existing approaches.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

arXiv AI

Jun 16

LaWAM: Latent World Action Models for Efficient Dynamics-Aware Robot Policies

arXiv:2606. 15768v1 Announce Type: cross Abstract: Vision-Language-Action models (VLAs) leverage large-scale vision-language pretraining for semantic robot control, but often lack explicit foresight into how robot actions change the scene.

By Jialei Chen, Kai Wang, Kang Chen, Shuaihang Chen, Feng Gao, Wenhao Tang, Zhiyuan Li, Weilin Liu, Zhuyu Yao, Boxun Li, Yuanbo Xu, Chao Yu

diffusion robotics multimodal benchmarks

arXiv AI

Jun 19

Finetuning Vision-Language-Action Models Requires Fewer Layers Than You Think

arXiv:2606. 20246v1 Announce Type: cross Abstract: Vision-Language-Action (VLA) models pre-trained on massive video-robot datasets have revolutionized robotic manipulation, yet their multi-billion parameter architectures impose prohibitive computational burdens during downstream fine-tuning and real-time inference.

By Gia-Binh Nguyen, Trong-Bao Ho, Thien-Loc Ha, Khoa Vo, Philip Lund M{\o}ller, Quang T. Nguyen, Long Dinh, Tuan Dam, Vu Duong, Tung M. Luu, Trung Le, Tran Nguyen Le, Minh Vu, An Thai Le, Ngan Le, Daniel Sonntag, James Zou, Jan Peters, Duy M. H. Nguyen, Ngo Anh Vien

robotics fine-tuning multimodal benchmarks safety

Hugging Face Trending Papers

Jul 27

DeVA: Decoupled Video-Action Model with physical guidance for robot policy learning

Generalizable robot manipulation requires policies that can anticipate how visual scenes evolve while executing language instructions. While recent Vision-Language-Action models benefit from large-scale pretraining, their predominantly static pretraining objectives provide limited supervision for physical dynamics and temporal causality, leaving control-relevant knowledge to be learned from downstream robot demonstrations.

robotics multimodal benchmarks

arXiv Computer Vision

6d ago

PDRVLA: A Vision-Language-Action Model with Physical Dynamics Representations

PDRVLA is a Vision‑Language‑Action model that incorporates a lightweight Physical Dynamics Module to extract physical tokens from vision‑language features, enabling action generation without online video synthesis. The module predicts future RGB, depth, and optical‑flow features, guided by auxiliary supervision from a frozen pretrained video world model. Trained solely on publicly available data augmented with human‑to‑robot synthesis, subtask annotation, and embodied chain‑of‑thought (ECoT) annotation, PDRVLA achieves high success rates across four simulation benchmarks, with ablations demonstrating the benefits of world‑model supervision and various conditioning strategies.

By Hanqing Wang, Zhaotian Cai, Ruichuan An, Bohan Zeng, Tao Zhang, Yilin Chen, Xu Jiang, Kaiquan Chen, Chongyang Wu, Haoran Li, Fan Yang, Mingan Lin, Wentao Zhang, Zenan Zhou, Youzheng Wu

diffusion robotics multimodal benchmarks

arXiv Machine Learning

Jun 16

Geometric Action Model for Robot Policy Learning

arXiv:2606. 17046v1 Announce Type: cross Abstract: Generalist robot policies must follow user instructions while reasoning about how objects, cameras, and robot actions interact in the 3D physical world.

By Jisang Han, Seonghu Jeon, Jaewoo Jung, Ren\'e Zurbr\"ugg, Honggyu An, Tifanny Portela, Marco Hutter, Marc Pollefeys, Seungryong Kim, Sunghwan Hong

robotics multimodal benchmarks

arXiv Computer Vision

Aug 27

Zero-WAM: In-Context World-Action Modeling from Human Videos for Open-Ended Task Generalization

Zero-WAM introduces a causal video-action model that enables robots to perform unseen manipulation tasks by following in-context human video guidance. The authors create HumanGen, a dataset of 74.2K human-robot ICL pairs across 8.6K tasks, and propose an in-context future chunk prediction objective to prevent shortcut learning. In simulation, Zero-WAM attains a 47.0% success rate on seven unseen tasks, outperforming the best video-action baseline by 29.5 percentage points, and demonstrates real‑world generalization to complex, long‑horizon, and fine‑grained tasks.

By Jiaming Zhou, Qihang Zhang, Gangwei Xu, Cunxin Fan, Yujie Zhao, Ruilin Wang, Yiming Luo, Shuai Yang, Xing Zhu, Yujun Shen, Junwei Liang, Yinghao Xu

llms robotics