EgoArgus is a new, human‑annotated dataset that tests visual‑language models (VLMs) as situational assistants in five everyday dialogue‑video scenarios. It evaluates how well VLMs understand and decide when to intervene, especially when visual and textual cues are helpful, irrelevant, or conflicting. The study finds that current VLMs still struggle to reliably act as egocentric assistants and that existing modality‑bias mitigation methods offer limited improvement.
arXiv:2610.12248v1 Announce Type: new Abstract: Wearable augmented reality (AR) assistants are moving toward continuous real-world interaction, where they perceive the user's activity through first-p...
The paper investigates when vision‑language models (VLMs) can independently analyze human‑centered video and when human oversight is still needed. By reviewing 1,702 CHI 2026 papers, the authors develop a five‑dimensional taxonomy of video annotation tasks and build a benchmark of 15 representative tasks. Experiments show that VLMs alone achieve near‑human accuracy (HNS = 97.0), while human verification of VLM outputs yields the highest accuracy (HNS = 121.5) and significantly reduces annotation time and cost.
arXiv:2609.39378v1 Announce Type: new Abstract: Real-world embodied tasks, from everyday activities to professional procedures, require agents to act under physical constraints while tracking evolvin...
arXiv:2607. 11523v1 Announce Type: cross Abstract: When should an intelligent assistant speak up without being asked?
BuddyVQA is a new benchmark for companion‑style question answering on egocentric streaming video, comprising 21.6K questions tied to 6K highlight moments across 1,012 long first‑person videos. It emphasizes two often overlooked aspects of daily first‑person QA: ego‑deictic expressions and interactively chained questions, requiring models to resolve visual pronouns and infer user intent within a long‑form streaming context. The authors propose MyBuddy, a multimodal chain‑of‑thought QA assistant that uses a question filter and multi‑level memory to efficiently retrieve visual and QA information, achieving significant performance gains on BuddyVQA and generalizing to other streaming and common video QA benchmarks.
When should an intelligent assistant speak up without being asked? Continuous egocentric video offers rich, evolving context that enables a new form of assistance: one that is proactive rather than merely reactive.
Video provides a rich record of human behavior, interaction, and situated contexts, offering important evidence for understanding people and conducting human-centered research. As vision-language mode...
arXiv:2604. 08342v2 Announce Type: replace Abstract: Long context egocentric video understanding has recently attracted significant research attention, with augmented reality (AR) highlighted as one of its most important application domains.
arXiv:2603. 09731v3 Announce Type: replace-cross Abstract: Multimodal large language models (MLLMs) are increasingly considered as a foundation for embodied agents, yet it remains unclear whether they can reliably reason about the long-term physical consequences of actions from an egocentric viewpoint.
arXiv:2608. 13113v1 Announce Type: cross Abstract: Recent advances in Multimodal Large Language Models (MLLMs) have led to substantial progress in video understanding, accompanied by a growing number of long video benchmarks.
arXiv:2606. 17627v1 Announce Type: cross Abstract: Fine-grained action recognition in egocentric video is challenging for Vision-Language Models (VLMs): actions often differ only in small visual cues, and a single model tends to be biased toward a subset of these cues.
The majority of our everyday activities are procedural and consist of sequences of interdependent steps. However, existing benchmarks for Visual Agents and Visual Language Models (VLMs) overlook the e...