EgoBench is an interactive egocentric multimodal benchmark designed to evaluate tool-using AI agents. It contains 1,590 video‑grounded tasks across five daily scenarios and a user‑agent‑tool environment that enforces joint visual perception, multi‑hop reasoning, and dynamic interaction. The benchmark includes a simulated user for realistic feedback and a deterministic validation framework, revealing that current video‑MLLM agents achieve only a 34.95% average Joint Success Rate, highlighting significant capability gaps.
arXiv:2606. 14397v1 Announce Type: new Abstract: As agentic systems continue to evolve and are widely deployed in real-world scenarios, there is a growing demand to faithfully evaluate their capabilities.
arXiv:2606.14397v4 Announce Type: replace Abstract: As agentic systems continue to evolve and are widely deployed in real-world scenarios, there is a growing demand to faithfully evaluate their capab...
arXiv:2609.39378v1 Announce Type: new Abstract: Real-world embodied tasks, from everyday activities to professional procedures, require agents to act under physical constraints while tracking evolvin...
AgentVidBench is a new multi‑hop video question‑answering benchmark designed to evaluate spatial, temporal, and causal reasoning in multimodal large language models (MLLMs). Unlike existing tests that focus on simple scene queries or global summaries, AgentVidBench includes step‑by‑step solution traces to assess whether agents gather the necessary evidence to justify their answers. Experiments with 12 MLLMs show limited single‑turn performance, but integrating these models into agentic workflows improves both accuracy and trajectory scores, establishing AgentVidBench as a comprehensive testbed for future research on agentic video understanding.
arXiv:2606. 28385v1 Announce Type: cross Abstract: Recent advances in robot world models enable synthetic video generation for embodied prediction and planning.
PlaySuite is a large-scale benchmark that evaluates interactive visual intelligence by using over 5,000 open-source video games from platforms like PyWeek and itch.io. The benchmark covers diverse game engines (Pygame, HTML5, Godot, Unity) and introduces a unified closed-loop interaction framework and a Video-LLM-as-a-judge protocol to standardize progress measurement. Evaluation of fourteen recent models shows a perception-action gap, with strong reasoning but poor sustained progress, spatial grounding, action execution, and self-correction.
arXiv:2609.15683v1 Announce Type: new Abstract: While In-Context Learning (ICL) enables models to adapt from exemplars without parameter updates, multimodal ICL remains largely underexplored, particu...
OmniAssistBench is a new benchmark for evaluating omni-modal large language models (Omni-LLMs) as real‑time video assistants that actively guide users toward goals. The benchmark addresses the challenge of dynamic interaction paths by providing models with predefined priors from source videos, forcing them to follow the same routes as users. The dataset was constructed by reverse‑engineering existing Internet videos into multi‑turn clips, a process that required over 1,000 expert person‑hours. Results show that proprietary Gemini‑3‑Pro scores 66.4/100 while open‑source Qwen3‑Omni‑Instruct scores 51.2, revealing that current models often give incorrect or incomplete answers, struggle with visual prompts, and fail to maintain context or delay responses until target events.
arXiv:2609.24526v2 Announce Type: replace Abstract: Physical AI requires models to ground visual and linguistic understanding in real-world environments while accounting for environmental constraints...
arXiv:2602. 22897v3 Announce Type: replace Abstract: Human intelligence naturally intertwines omni-modal perception -- spanning vision, audio, and language -- with complex reasoning and tool usage to interact with the world.
Ego2Act is a new benchmark that tests video generation models on goal‑directed, egocentric manipulation tasks. It contains 2,640 videos from 110 real‑world tasks, each requiring multiple steps of object manipulation to achieve a high‑level goal. The benchmark includes Ego2ActJudge, a reference‑free evaluation pipeline that better aligns with human judgments of task completion and physics plausibility.
SceneTeract is a verification interface that separates semantic action understanding from physical feasibility in indoor 3D scenes. It decomposes activities into atomic actions and performs explicit geometric checks to determine executability, providing diagnostic traces for failures. The system reveals widespread functional and accessibility issues in synthetic scenes, shows that existing VLMs over‑predict action feasibility, and improves VLM performance through post‑training with verifier feedback, with benefits that generalize to real‑world scenes.