Omni-InteractScope is a new benchmark comprising 1,479 human‑annotated interactions and 1,790 temporal targets across eight task groups, designed to evaluate how well omnimodal large language models (Omni‑LLMs) connect continuous audiovisual streams to user needs and deliver timely, sustained interventions. The benchmark links input modalities, event triggers, intervention types, and user needs, covering event‑timed tool invocation, query‑free assistance, notifications, warnings, state tracking, and continuous responses. Experimental results on this benchmark reveal three critical limitations of current Omni‑LLMs: unreliable timely tool invocation, poor coverage of query‑free reminders, and limited end‑to‑end guidance coverage.
whyItMatters":"The benchmark highlights specific performance gaps in Omni‑LLMs that are essential for developing streaming assistants capable of timely action, proactive initiative, and comprehensive response coverage."
By Yinsong Yan, Ling Wang, Yuxuan Wang, Yifan Yang, Ruiyang Xu, Junming Lin, Qi Chen, Meng Gao, Minghao Han, Yunfei Chu, Qize Yang, Xize Cheng, Jin Xu, Jibin Wu
The paper introduces “P…IBench”, a benchmark for proactive video understanding that tests daily perception, risk prediction, and intervention in real‑life streaming scenarios. It evaluates 12 video‑language models, finding that current systems struggle to make well‑calibrated response decisions. The authors also create new proactive training data and train “P…I-Assistant-4B”, which shows uneven improvements across the benchmark’s capabilities.
By Boyu Cui, Junhao Luo, Zhihao Wang, Linghan Chen, Zejun Li, Jiwen Zhang, Siyuan Wang, Zhongyu Wei
OmniAssistBench is a new benchmark for evaluating omni-modal large language models (Omni-LLMs) as real‑time video assistants that actively guide users toward goals. The benchmark addresses the challenge of dynamic interaction paths by providing models with predefined priors from source videos, forcing them to follow the same routes as users. The dataset was constructed by reverse‑engineering existing Internet videos into multi‑turn clips, a process that required over 1,000 expert person‑hours. Results show that proprietary Gemini‑3‑Pro scores 66.4/100 while open‑source Qwen3‑Omni‑Instruct scores 51.2, revealing that current models often give incorrect or incomplete answers, struggle with visual prompts, and fail to maintain context or delay responses until target events.
By Xianyun Sun, Chaoyou Fu, Zhengye Zhang, Feiyang Duan, Qingyuan Cao, Yonghui Niu, Sihang Yuan, Ge Zhang, Caifeng Shan
EgoVoice is a framework designed to train and evaluate proactive spoken assistants that operate from egocentric multimodal streams. By processing first-person video and audio from HoloAssist recordings, the system creates clean audio streams and frames each moment as a decision to either stay silent or provide spoken guidance. Experiments show that EgoVoice improves intervention timing, content relevance, and human preference compared to existing zero‑shot models.
By Heeseung Kim
Natural interaction in digital and physical environments requires continuous perception and timely responses. Spoken dialogue relies on acoustic and linguistic cues, while video interaction also requi...
arXiv:2610.28402v1 Announce Type: new
Abstract: As streaming video generation advances, enabling real-time interaction with generated video is becoming increasingly important. We introduce StreamEval...
By StreamEval Team