arXiv AI

AFFORD2ACT: How Far Can Sparse Keypoints Go for Robot Manipulation?

AFFORD2ACT investigates whether robot manipulation policies can rely on a very compact visual representation instead of dense images. Using an affordance model to select up to 15 object keypoints and 4 end‑effector keypoints, the policy receives only 19 2D points (38 coordinates). In six real‑robot tasks, this sparse keypoint input outperforms RGB, RGB‑D, and dense‑affordance baselines, maintains performance with fewer demonstrations, and achieves 82% success on held‑out data, though it depends on accurate affordance localization and keypoint tracking.

arXiv Computer Vision

Aug 27

Training-Free Interaction-Aligned Visual Token Pruning for Efficient Embodied Manipulation

The paper introduces Interaction‑Aligned Pruning (IAprune), a training‑free method for visual token pruning in embodied manipulation tasks. IAprune jointly decides per‑frame budget and token selection, using semantic‑motion spatial agreement to choose between conservative and aggressive coverage, and applies geometric residual correction to focus on under‑represented boundaries. Experiments on four policies, three simulation benchmarks, and a real‑robot platform show that IAprune matches unpruned performance on LIBERO while achieving up to 1.54× speed‑up and 1.48× acceleration on a real robot.

By Jintao Cheng, Weibin Li, Haozhe Wang, Gang Wang, Yipu Zhang, Xiaoyu Tang, Jin Wu, Xieyuanli Chen, Yunhui Liu, Wei Zhang

robotics efficiency multimodal benchmarks

arXiv Computer Vision

Sep 22

AffordanceWAM: Affordance-Aware Joint World-Action Modeling for Robot Manipulation

arXiv:2609.22332v1 Announce Type: cross Abstract: Generalizable robot manipulation requires predicting how a scene will evolve, identifying where interactions are feasible, and determining how to act...

By Jiadi You, Qize Yu, Yue Chen, Minghong Cai, Zhide Zhong, Yuran Wang, Bowen Ping, Jiaqi Liang, Zhenhao Shen, Haodong Yan, Yinchuan Li, Ruihai Wu, Xiaojuan Qi, Yingcong Chen

llms diffusion robotics multimodal

arXiv Computer Vision

Oct 7

VLA-ACL: Action-Consistent Visual Token Pruning for Efficient Vision-Language-Action Models

arXiv:2610.08133v1 Announce Type: cross Abstract: Vision-Language-Action (VLA) models achieve strong robotic manipulation performance but incur high computational costs from processing long token seq...

By Owen Du, Yang Yue, Jie Zhang, Jiaqi Pi, Chi Bene Chen, Gao Huang

robotics fine-tuning efficiency multimodal

arXiv Machine Learning

Jul 21

Foresight Residual RL for Long-Horizon Robot Manipulation with Vision-Language-Action Models

arXiv:2607. 16506v1 Announce Type: cross Abstract: Vision-Language-Action (VLA) policies offer strong general-purpose manipulation priors, but often fail on tight-tolerance, contact-rich assembly due to long-horizon credit assignment and subtask coupling: a state that is geometrically successful for the current skill can be brittle for downstream skills.

By Yuhan Liu, Xinyu Zhang, Litao Liu, Abdeslam Boularias

reinforcement-learning robotics multimodal

Hugging Face Trending Papers

Jul 27

DeVA: Decoupled Video-Action Model with physical guidance for robot policy learning

Generalizable robot manipulation requires policies that can anticipate how visual scenes evolve while executing language instructions. While recent Vision-Language-Action models benefit from large-scale pretraining, their predominantly static pretraining objectives provide limited supervision for physical dynamics and temporal causality, leaving control-relevant knowledge to be learned from downstream robot demonstrations.

robotics multimodal benchmarks

arXiv Machine Learning

Jun 10

Dexterous Point Policy: Learning Point-based Dexterous Hand Policies from Human Demonstrations

arXiv:2606. 10614v1 Announce Type: cross Abstract: Robotic foundation models pre-trained on human demonstration videos have shown promise, but a significant embodiment gap remains when the resulting policies are deployed on real robots.

By Beomjun Kim, Seong Hyeon Park, Seunghoon Sim, Seungjun Moon, Sanghyeok Lee, Jinwoo Shin

llms agents robotics fine-tuning benchmarks

arXiv Machine Learning

Jun 10

GHOST: Hierarchical Sub-Goal Policies for Generalizing Robot Manipulation

arXiv:2606. 10025v1 Announce Type: cross Abstract: We present GHOST, a framework for learning visuomotor manipulation policies that generalize beyond the training distribution.

By Sriram Krishna, Ben Eisner, Haotian Zhan, Ying Yuan, Haoyu Zhen, Chuang Gan, Shubham Tulsiani, David Held

diffusion robotics

arXiv AI

Jun 2

AffordGen: Generating Diverse Demonstrations for Generalizable Object Manipulation with Afford Correspondence

arXiv:2604. 10579v2 Announce Type: replace-cross Abstract: Despite the recent success of modern imitation learning methods in robot manipulation, their performance is often constrained by geometric variations due to limited data diversity.

By Jiawei Zhang, Kaizhe Hu, Yingqian Huang, Yuanchen Ju, Zhengrong Xue, Huazhe Xu

robotics

arXiv Computer Vision

Sep 30

WorldLine: Action-Driven Visual Simulation for Robotic Manipulation

arXiv:2609.38059v1 Announce Type: cross Abstract: Real-world robot learning is constrained by the cost of collecting experience and evaluating candidate behaviors. Video generation models offer a sca...

By Shenghe Zheng, Wenbo Li, Jiyao Zhang, Bin Xia, Haoyang Huang, Nan Duan, Jiaya Jia

diffusion robotics efficiency

arXiv AI

Jun 30

Grounding Sim-to-Real Generalization in Robotic Manipulation: An Empirical Study with Vision-Language-Action Models

arXiv:2603. 22876v2 Announce Type: replace-cross Abstract: Learning a generalist control policy for robotic manipulation typically relies on large-scale datasets.

By Ruixing Jin, Zicheng Zhu, Ruixiang Ouyang, Sheng Xu, Bo Yue, Zhizheng Wu, Guiliang Liu

reinforcement-learning robotics multimodal benchmarks

arXiv Computer Vision

Oct 9

Dex-One2Many: Learning Dexterous Manipulation from a Single Human Demonstration

Dex-One2Many introduces a real-to-sim-to-real framework that learns a generalizable dexterous manipulation policy from a single human video. By abstracting the video into sequential scene graphs, the method guides reinforcement learning with dense rewards and diverse reset states, enabling efficient exploration while preserving broad generalizability. Trained entirely in simulation, Dex-One2Many achieves zero‑shot transfer to a real multi‑fingered hand and outperforms baselines by 6.5% on seen configurations and 71% on unseen scenarios across five tool‑use and manipulation tasks.

By Jusuk Lee, Sungha Kim, Yeonsoo Park, Jonguk Cheon, Yoonkyo Jung, Yongjun You, H. Jin Kim, Jia-Bin Huang, Furong Huang, Youngseok Jang, Seungjae Lee

reinforcement-learning robotics

arXiv AI

Sep 18

HIL-UMI: Bringing Human-in-the-Loop Post-Training of Vision-Language-Action Models to Universal Manipulation Interface

HIL-UMI is a policy-guided Universal Manipulation Interface that enables robot‑free, human‑in‑the‑loop post‑training of vision‑language‑action models. By querying the current policy during handheld demonstrations and using an Energy Score to detect out‑of‑distribution states, it selectively collects new data and refines a progress‑based advantage estimator. The updated estimator then drives advantage‑conditioned behavioral cloning, improving performance on long‑horizon and precise manipulation tasks while reducing per‑frame collection time compared to HG‑DAgger.

By Zimu Han, Yiming Zeng, Jiyao Zhang, Zihao Zhao, Yuanfei Wang, Yixiang Jin, Shiqi Li, Shuangben Chen, Wei Huang, Ruodai Li, Hui Shen, Hao Dong

robotics fine-tuning multimodal

Related stories