Omni-InteractScope is a new benchmark comprising 1,479 human‑annotated interactions and 1,790 temporal targets across eight task groups, designed to evaluate how well omnimodal large language models (Omni‑LLMs) connect continuous audiovisual streams to user needs and deliver timely, sustained interventions. The benchmark links input modalities, event triggers, intervention types, and user needs, covering event‑timed tool invocation, query‑free assistance, notifications, warnings, state tracking, and continuous responses. Experimental results on this benchmark reveal three critical limitations of current Omni‑LLMs: unreliable timely tool invocation, poor coverage of query‑free reminders, and limited end‑to‑end guidance coverage.
whyItMatters":"The benchmark highlights specific performance gaps in Omni‑LLMs that are essential for developing streaming assistants capable of timely action, proactive initiative, and comprehensive response coverage."
By Yinsong Yan, Ling Wang, Yuxuan Wang, Yifan Yang, Ruiyang Xu, Junming Lin, Qi Chen, Meng Gao, Minghao Han, Yunfei Chu, Qize Yang, Xize Cheng, Jin Xu, Jibin Wu
The paper introduces “P…IBench”, a benchmark for proactive video understanding that tests daily perception, risk prediction, and intervention in real‑life streaming scenarios. It evaluates 12 video‑language models, finding that current systems struggle to make well‑calibrated response decisions. The authors also create new proactive training data and train “P…I-Assistant-4B”, which shows uneven improvements across the benchmark’s capabilities.
By Boyu Cui, Junhao Luo, Zhihao Wang, Linghan Chen, Zejun Li, Jiwen Zhang, Siyuan Wang, Zhongyu Wei
OmniAssistBench is a new benchmark for evaluating omni-modal large language models (Omni-LLMs) as real‑time video assistants that actively guide users toward goals. The benchmark addresses the challenge of dynamic interaction paths by providing models with predefined priors from source videos, forcing them to follow the same routes as users. The dataset was constructed by reverse‑engineering existing Internet videos into multi‑turn clips, a process that required over 1,000 expert person‑hours. Results show that proprietary Gemini‑3‑Pro scores 66.4/100 while open‑source Qwen3‑Omni‑Instruct scores 51.2, revealing that current models often give incorrect or incomplete answers, struggle with visual prompts, and fail to maintain context or delay responses until target events.
By Xianyun Sun, Chaoyou Fu, Zhengye Zhang, Feiyang Duan, Qingyuan Cao, Yonghui Niu, Sihang Yuan, Ge Zhang, Caifeng Shan
EgoVoice is a framework designed to train and evaluate proactive spoken assistants that operate from egocentric multimodal streams. By processing first-person video and audio from HoloAssist recordings, the system creates clean audio streams and frames each moment as a decision to either stay silent or provide spoken guidance. Experiments show that EgoVoice improves intervention timing, content relevance, and human preference compared to existing zero‑shot models.
By Heeseung Kim
Natural interaction in digital and physical environments requires continuous perception and timely responses. Spoken dialogue relies on acoustic and linguistic cues, while video interaction also requi...
arXiv:2610.28402v1 Announce Type: new
Abstract: As streaming video generation advances, enabling real-time interaction with generated video is becoming increasingly important. We introduce StreamEval...
By StreamEval Team
arXiv:2609. 27303v1 Announce Type: new Abstract: Livestreams are long-lasting interactive environments where audiovisual content, viewer activity, host behavior, and platform signals evolve together, creating assistance needs that emerge from the stream itself.
By Shujian Gao, Jiamei Yan, Yuchen Yang, Penghao Zhou, Qinglei Wang, Tiehan Fan, Yuan Wang, Zuxuan Wu, Yu-gang Jiang
arXiv:2609.13814v1 Announce Type: new
Abstract: Natural interaction in digital and physical environments requires continuous perception and timely responses. Spoken dialogue relies on acoustic and li...
By Ruixiang Zhao, Hualei Wang, Renhe Sun, Enzhi Zhou, Jincenzi Wu, Xujie Song, Kexin Shi, Zihang Liu, Pengcheng Zhu, Jiayi Zhou, Baoyue Zhang, Changhao Zhang, Zitong Wang, Jinhong Wang, Tong Niu, Jingjing Liu, Junan Lin, Haolin He, Hengshuo Chu, Yuhui Chen, Jian Liu, Yuge Huang, Junliang Xing, Yuntao Wang, Weiqiang Wang, Chun Yu, Yuanchun Shi
LiveProBench evaluates streaming video models on their ability to interact proactively, assessing whether they respond at appropriate times without explicit cues. The benchmark tests models at one‑second intervals across six subtasks that vary trigger ambiguity and timing tolerance, measuring response accuracy, silence rates, and duplicate responses. Results show that many models issue premature responses more often than missed ones, highlighting a significant shortfall in human‑like temporal decision making.
By Kaixuan Du, Xin Wan, Hang Zhang, Meng Cao, Dai Guan, Ming Chen, YuKun Wang
Today's multimodal assistants are typically prompted to interpret the visual evidence presented to them. To offload the cognitive burden of remembering across long horizons, wearable assistants should...
OVIBench introduces the first standardized benchmark for evaluating vision‑language models on Online Video Question Answering under Interruption, a realistic setting where users can interrupt the model during answer generation. The benchmark categorizes interruptions into Cancellation, False Trigger, and Correction, supports both open‑ended and multiple‑choice tasks, and provides an offline simulation protocol plus a multi‑dimensional metric suite. Experiments show that OVIBench can distinguish models’ interruption‑handling abilities, particularly in following correction requests, and that fine‑tuning on the newly created OVI‑Train dataset yields significant performance gains.
By Naiming Liu, Zhiheng Wu, Shuning Wang, Tie Zhang, Bowen Liu, Tong Wang
ProactiveBench evaluates streaming video models on their ability to interact proactively, rather than reactively. It tests models at one‑second intervals without explicit cues, using six subtasks that vary trigger ambiguity, timing tolerance, and response patterns. The benchmark measures both response and silence rates, distinguishing early, in‑window, and missed responses, and penalizes omissions and repetitions.
By Kaixuan Du, Xin Wan, YuKun Wang, Hang Zhang, Meng Cao, Dai Guan, Ming Chen, Ni Li