arXiv Computer Vision

Egocentric Action Recognition with Objects and Vision-Language Affordances

The paper introduces a method for egocentric action recognition that fuses a frozen SlowFast video model with an attention-based object expert. Each object is represented by affordances generated zero‑shot from a vision‑language model, and an interaction affordance filters out non‑active detections. On the MECCANO dataset, this approach improves Top‑1 accuracy from 44.9% (video only) to 49.9% when both affordances are used, surpassing existing baselines.

arXiv Computer Vision
Oct 1

EgoTools: Towards Tool-Centric Reasoning in Real-World Egocentric Videos

arXiv:2609.39378v1 Announce Type: new Abstract: Real-world embodied tasks, from everyday activities to professional procedures, require agents to act under physical constraints while tracking evolvin...

By Shulin Tian, Junsu Kim, Shuai Liu, Hao Li, Yujiao Shen, Sihan Li, Zhe Yang, Yeongon Kim, Feiyu Li, Jialin Wu, Yichi Zhang, Wenhui Wang, Runmao Yao, Yuhao Dong, Zhaoxi Chen, Fangzhou Hong, Antonino Furnari, Jingkang Yang, Hongyuan Zhu, Ziwei Liu
arXiv Computer Vision
Nov 5

EgoM2P: Egocentric Multimodal Multitask Pretraining

EgoM2P is a masked‑modeling framework that learns from temporally‑aware multimodal tokens to train a large, general‑purpose model for egocentric 4D understanding. It supports multitasking across diverse egocentric perception and synthesis tasks—such as gaze prediction, egocentric camera tracking, monocular depth estimation, and conditional egocentric video synthesis—while matching or surpassing specialist models at a fraction of the computational cost. The authors plan to fully open‑source EgoM2P to advance egocentric vision research.

By Gen Li, Yutong Chen, Yiqian Wu, Kaifeng Zhao, Marc Pollefeys, Siyu Tang
Hugging Face Trending Papers
Jul 9

Do Egocentric Video-Language Models Capture Both Hand- and Object-Centric Cues?

Hand-object interaction (HOI) recognition requires capturing both hand manipulations and object transformations. However, existing video-language models often fall into shortcuts by relying on spurious correlations among hands, objects, or environmental context, rather than reasoning from the appearance and dynamics of hands and objects themselves.

arXiv AI
Jun 29

EXPLORE-Bench: Egocentric Scene Prediction with Long-Horizon Reasoning

arXiv:2603. 09731v3 Announce Type: replace-cross Abstract: Multimodal large language models (MLLMs) are increasingly considered as a foundation for embodied agents, yet it remains unclear whether they can reliably reason about the long-term physical consequences of actions from an egocentric viewpoint.

By Chengjun Yu, Xuhan Zhu, Chaoqun Du, Pengfei Yu, Wei Zhai, Yang Cao, Zheng-Jun Zha
arXiv Computer Vision
Aug 27

Moving Beyond More Views: Redundancy-Aware Ego-Exo Fusion for Proficiency Estimation

The paper introduces a redundancy-aware fusion framework for EgoExo proficiency estimation, which integrates fine-grained motion cues from egocentric views with spatial context from exocentric views. It identifies multiview redundancy and overfitting as key challenges and proposes two modules—AdaMVS for adaptive view selection and VIB-GB for compressing redundant signals—to address them. Experiments on EgoExo-4D and EgoExo-Fitness show that the method learns to select informative views and fuse them effectively, achieving state‑of‑the‑art results.

By Xu Dong, Wanqing Li, Anthony Adeyemi-Ejeye, Andrew Gilbert
arXiv Computer Vision
Sep 22

AffordanceWAM: Affordance-Aware Joint World-Action Modeling for Robot Manipulation

arXiv:2609.22332v1 Announce Type: cross Abstract: Generalizable robot manipulation requires predicting how a scene will evolve, identifying where interactions are feasible, and determining how to act...

By Jiadi You, Qize Yu, Yue Chen, Minghong Cai, Zhide Zhong, Yuran Wang, Bowen Ping, Jiaqi Liang, Zhenhao Shen, Haodong Yan, Yinchuan Li, Ruihai Wu, Xiaojuan Qi, Yingcong Chen