arXiv Computer Vision

EgoTools: Towards Tool-Centric Reasoning in Real-World Egocentric Videos

arXiv AI
Jun 29

EXPLORE-Bench: Egocentric Scene Prediction with Long-Horizon Reasoning

arXiv:2603. 09731v3 Announce Type: replace-cross Abstract: Multimodal large language models (MLLMs) are increasingly considered as a foundation for embodied agents, yet it remains unclear whether they can reliably reason about the long-term physical consequences of actions from an egocentric viewpoint.

By Chengjun Yu, Xuhan Zhu, Chaoqun Du, Pengfei Yu, Wei Zhai, Yang Cao, Zheng-Jun Zha
Hugging Face Trending Papers
Nov 2

EgoActionBench: Benchmarking Dense Action Captioning for Egocentric Human Videos

EgoActionBench is a new benchmark for dense action captioning in egocentric videos, featuring 1,407 episodes and 41,132 fine‑grained action annotations that specify which hand (left, right, or both) performs each temporally localized atomic action. The benchmark introduces an evaluation protocol that assesses semantic fidelity, temporal overlap, action precision, and recall, thereby measuring what happens, when it happens, and which hand is involved. Twenty baseline models were evaluated, and two backbones (Qwen3‑VL‑4B and LLaVA‑OneVision‑2‑8B) were fine‑tuned and reinforced, achieving significant score improvements and outperforming all open‑source baselines.

arXiv AI
Oct 28

SOMA: A Benchmark for Fine-Grained Egocentric Behavior and Body-Motion Understanding

SOMA is a new benchmark that tests fine‑grained egocentric body‑motion understanding, featuring 1,632 human‑verified questions across body configuration, motion dynamics, and temporal reasoning. The study evaluates 37 multimodal large language models (MLLMs) and finds that humans achieve 68.5% accuracy while the best MLLM (Qwen3.6‑27B) reaches only 50.0% zero‑shot accuracy. The authors also provide a training dataset and a multimodal baseline to help close the performance gap and highlight common failure modes such as misreading body parts and temporal errors.

By Matea Tashkovska, Na\"el Ouerghemi, Haozhe Qi, Alexander Mathis
arXiv Computer Vision
Nov 6

Egocentric Action Recognition with Objects and Vision-Language Affordances

The paper introduces a method for egocentric action recognition that fuses a frozen SlowFast video model with an attention-based object expert. Each object is represented by affordances generated zero‑shot from a vision‑language model, and an interaction affordance filters out non‑active detections. On the MECCANO dataset, this approach improves Top‑1 accuracy from 44.9% (video only) to 49.9% when both affordances are used, surpassing existing baselines.

By Ra\'ul Boix-Aldeguer, Manuel Benavent-Lledo, David Ortiz-Perez, Jose Garcia-Rodriguez
arXiv Computer Vision
Oct 2

Ego2Act: Evaluating Goal-Directed Manipulation in Egocentric Video Generation

Ego2Act is a new benchmark that tests video generation models on goal‑directed, egocentric manipulation tasks. It contains 2,640 videos from 110 real‑world tasks, each requiring multiple steps of object manipulation to achieve a high‑level goal. The benchmark includes Ego2ActJudge, a reference‑free evaluation pipeline that better aligns with human judgments of task completion and physics plausibility.

By Patrick Amadeus Irawan, Iskandar Muda Rizky Parlambang, Rava Maulana, Qinrong Cui, Erland Hilman Fuadi, Zayd M. K. Zuhri, Nanda Ryaas Absar, Ahmed Elshabrawy, Wilfried Ariel Mulyawan, Shoubin Yu, Yue Zhang, Mohit Bansal, Alham Fikri Aji
arXiv AI
Oct 9

EgoBench: An Interactive Egocentric Multimodal Benchmark for Tool-Using Agents

EgoBench is an interactive egocentric multimodal benchmark designed to evaluate tool-using AI agents. It contains 1,590 video‑grounded tasks across five daily scenarios and a user‑agent‑tool environment that enforces joint visual perception, multi‑hop reasoning, and dynamic interaction. The benchmark includes a simulated user for realistic feedback and a deterministic validation framework, revealing that current video‑MLLM agents achieve only a 34.95% average Joint Success Rate, highlighting significant capability gaps.

By Yunqi Liu, Tong Niu, Zitong Wang, Zhenlong Dai, Yuqi Qing, Weiqiang Wang, Jian Liu
arXiv AI
Aug 20

EgoMemReason: A Memory-Driven Reasoning Benchmark for Long-Horizon Egocentric Video Understanding

EgoMemReason is a new benchmark for week‑long egocentric video understanding that focuses on memory‑driven reasoning rather than simple perception tasks. It tests three memory types—entity, event, and behavior—across 500 questions, each requiring evidence from an average of 5.1 video segments and 25.9 hours of backtracking. Evaluation of 17 models shows that even the best achieves only 39.6% accuracy, highlighting the difficulty of long‑horizon memory in multimodal systems.

By Ziyang Wang, Yue Zhang, Shoubin Yu, Ce Zhang, Zengqi Zhao, Jaehong Yoon, Hyunji Lee, Gedas Bertasius, Mohit Bansal
arXiv Computation and Language
Aug 27

EgoArgus: Benchmarking VLMs as Situational Assistants for Modality-Grounded User Supports

EgoArgus is a new, human‑annotated dataset that tests visual‑language models (VLMs) as situational assistants in five everyday dialogue‑video scenarios. It evaluates how well VLMs understand and decide when to intervene, especially when visual and textual cues are helpful, irrelevant, or conflicting. The study finds that current VLMs still struggle to reliably act as egocentric assistants and that existing modality‑bias mitigation methods offer limited improvement.

By Yu-Chien Tang, Yu-Hsiang Liu, An-Zi Yen