EgoTools: Towards Tool-Centric Reasoning in Real-World Egocentric Videos
Read the original on arXiv Computer Vision →The Flow has not summarised this story yet — read it at arXiv Computer Vision.
The Flow has not summarised this story yet — read it at arXiv Computer Vision.
arXiv:2610.18499v1 Announce Type: cross Abstract: Egocentric video understanding spans diverse capabilities, modalities, and evaluation formats, yet these are often studied in isolation despite their...
arXiv:2603. 09731v3 Announce Type: replace-cross Abstract: Multimodal large language models (MLLMs) are increasingly considered as a foundation for embodied agents, yet it remains unclear whether they can reliably reason about the long-term physical consequences of actions from an egocentric viewpoint.
EgoActionBench is a new benchmark for dense action captioning in egocentric videos, featuring 1,407 episodes and 41,132 fine‑grained action annotations that specify which hand (left, right, or both) performs each temporally localized atomic action. The benchmark introduces an evaluation protocol that assesses semantic fidelity, temporal overlap, action precision, and recall, thereby measuring what happens, when it happens, and which hand is involved. Twenty baseline models were evaluated, and two backbones (Qwen3‑VL‑4B and LLaVA‑OneVision‑2‑8B) were fine‑tuned and reinforced, achieving significant score improvements and outperforming all open‑source baselines.
SOMA is a new benchmark that tests fine‑grained egocentric body‑motion understanding, featuring 1,632 human‑verified questions across body configuration, motion dynamics, and temporal reasoning. The study evaluates 37 multimodal large language models (MLLMs) and finds that humans achieve 68.5% accuracy while the best MLLM (Qwen3.6‑27B) reaches only 50.0% zero‑shot accuracy. The authors also provide a training dataset and a multimodal baseline to help close the performance gap and highlight common failure modes such as misreading body parts and temporal errors.
The paper introduces a method for egocentric action recognition that fuses a frozen SlowFast video model with an attention-based object expert. Each object is represented by affordances generated zero‑shot from a vision‑language model, and an interaction affordance filters out non‑active detections. On the MECCANO dataset, this approach improves Top‑1 accuracy from 44.9% (video only) to 49.9% when both affordances are used, surpassing existing baselines.
Ego2Act is a new benchmark that tests video generation models on goal‑directed, egocentric manipulation tasks. It contains 2,640 videos from 110 real‑world tasks, each requiring multiple steps of object manipulation to achieve a high‑level goal. The benchmark includes Ego2ActJudge, a reference‑free evaluation pipeline that better aligns with human judgments of task completion and physics plausibility.