EgoHSI3R is a unified feed‑forward framework that reconstructs 4D hand‑scene interactions from egocentric monocular video. It jointly estimates camera motion, dense scene geometry, world‑space bilateral MANO hands, and per‑vertex hand contact by maintaining a recurrent scene representation and using hand‑centric prompts and MANO vertex queries. The method achieves the lowest world‑space hand joint error on ARCTIC while offering faster inference than existing pipelines.
By Mingzhi Pei, Zhangchen Ye, Ruoqu Chen, Liu Cao, Mengdi Xu
MINT is a foundation model that directly predicts world-space two-hand trajectories from egocentric RGB video, jointly estimating camera motion, hand states, and hand presence in a single spatiotemporal representation. It uses an open-source labeling pipeline, EGOPIPELINE, to generate large-scale pseudo-labels for pretraining, followed by fine-tuning on a small set of high-quality joint annotations. The model outperforms existing multi-stage approaches in accuracy and speed, and generalizes zero‑shot to unseen egocentric datasets.
By Zijie Zhu, Weiren Cai, Yizhou Wang, Zhenjie Yang, Yide Liu, Jiahao Chen, Guanqi He
arXiv:2610.29936v1 Announce Type: new
Abstract: Recovering metric world-space hand motion from egocentric video supports understanding human interactions and transferring them to embodied agents. Thr...
By Naru Suzuki, Satoshi Ikehata, Takehiko Ohkawa, Yoichi Sato, Shuhei Kurita
arXiv:2609.38615v1 Announce Type: cross
Abstract: Egocentric videos of human manipulation provide valuable visual experience for embodied intelligence, yet collecting such data at scale is costly. Ex...
By Hongjia Zhai, Xiyu Zhang, Haoran Zhang, Zhichao Ye, Haomin Liu, Guofeng Zhang, Ian Reid, Xingxing Zuo
The paper introduces P³-WAM, a unified world action model for egocentric dexterous manipulation that integrates 3D perception, action-conditioned prediction, and goal-directed planning. It operates in three modes—Perceive, Predict, and Plan—using a mixture-of-transformers architecture that shares base weights and adapts to modality-specific inputs. Experiments demonstrate that unified training improves performance across all three capabilities, outperforming specialized baselines on 3D hand reconstruction and action-conditioned video prediction while producing more instruction‑adherent manipulation videos.
By Zirui Wu, Chen Guo, Tianjian Jiang, Zeren Jiang, Zeming Li, Lingchen Yang, Longteng Duan, Ying Xue, Handi Yin, Shenyuan Gao, Martin R. Oswald, Jun Zhang, Ping Tan, Jie Song
arXiv:2608.20308v2 Announce Type: replace
Abstract: Egocentric video offers scalable manipulation data for embodied AI, yet recovering metric 3D hand trajectories remains challenging due to severe ob...
By Yufei Liu, Xixi Wang, Hao Li, Ganlong Zhao, Kaitong Cai, Chengkai Jin, Chunxiao Liu, Jianbo Liu, Siyuan Huang, Xingang Pan, Hongsheng Li