EgoErrorVQA: Assess Egocentric Comprehension Capabilities through Procedural Errors for Ego-Agentic AI
Read the original on Hugging Face Trending Papers →The Flow has not summarised this story yet — read it at Hugging Face Trending Papers.
The Flow has not summarised this story yet — read it at Hugging Face Trending Papers.
EgoErrorVQA introduces a new egocentric visual question answering task that evaluates visual agents’ ability to detect procedural errors in everyday activities. The paper presents an evaluator agent built on the Agent2Agent protocol and shows that current models struggle with procedural error recognition. It also proposes Ego-ADR, an Adaptive Decoupled Reasoning framework that improves performance on the task, achieving state‑of‑the‑art results.
arXiv:2607. 24770v1 Announce Type: new Abstract: Procedural tasks such as furniture assembly and home repair impose substantial cognitive demands because users must interpret instructions, track task progress, reason about spatial state, and recover from errors while performing physical actions.
ReferClaw is an agentic framework designed to improve egocentric AI assistants by treating question answering as an interactive process. It uses three cooperating agents to generate candidate interpretations, ask clarifying questions, and produce answers grounded in resolved intent and visual evidence. The system balances information gain against user effort, learns from interaction, and shows higher accuracy and joint answer‑grounding performance on augmented egocentric QA benchmarks, with zero‑shot transfer across datasets.
EgoArgus is a new, human‑annotated dataset that tests visual‑language models (VLMs) as situational assistants in five everyday dialogue‑video scenarios. It evaluates how well VLMs understand and decide when to intervene, especially when visual and textual cues are helpful, irrelevant, or conflicting. The study finds that current VLMs still struggle to reliably act as egocentric assistants and that existing modality‑bias mitigation methods offer limited improvement.
ReferClaw is an agentic framework designed to improve egocentric AI assistants by treating question answering as an interactive process. It uses three cooperating agents to generate candidate interpretations, ask clarifying questions, and produce answers grounded in resolved intent and visual evidence. The system balances information gain against user effort, learns from interaction, and shows better accuracy and grounding on augmented egocentric QA benchmarks, including zero‑shot transfer across datasets.
The paper introduces Vision-as-a-Tool (VaaT), a paradigm that treats multimodal large language models as external utilities for visual reasoning, allowing text-only agents to offload visual tasks and vision-capable agents to augment perception on demand. It presents VHarness, a unified framework with five atomic actions, and evaluates it on Image-to-HTML, Image-to-SVG, and Image-to-3D generation tasks. Results show consistent improvements in visual fidelity and geometric accuracy for both text-only and multimodal coding agents.