← Back to all news
arXiv AI By Yinsong Yan, Ling Wang, Yuxuan Wang, Yifan Yang, Ruiyang Xu, Junming Lin, Qi Chen, Meng Gao, Minghao Han, Yunfei Chu, Qize Yang, Xize Cheng, Jin Xu, Jibin Wu

Omni-InteractScope: Benchmarking Proactive and Agentic Interaction in Continuous Omnimodal Streams

Read the original on arXiv AI →

Omni-InteractScope is a new benchmark comprising 1,479 human‑annotated interactions and 1,790 temporal targets across eight task groups, designed to evaluate how well omnimodal large language models (Omni‑LLMs) connect continuous audiovisual streams to user needs and deliver timely, sustained interventions. The benchmark links input modalities, event triggers, intervention types, and user needs, covering event‑timed tool invocation, query‑free assistance, notifications, warnings, state tracking, and continuous responses. Experimental results on this benchmark reveal three critical limitations of current Omni‑LLMs: unreliable timely tool invocation, poor coverage of query‑free reminders, and limited end‑to‑end guidance coverage. whyItMatters":"The benchmark highlights specific performance gaps in Omni‑LLMs that are essential for developing streaming assistants capable of timely action, proactive initiative, and comprehensive response coverage."

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

Related stories

arXiv Computation and Language
3d ago

Omni-Life: Benchmarking Omnimodal Real-Time AI Assistants in People's Daily Lives

Omni-Life is a new benchmark for real‑time omnimodal AI assistants, built around real‑world user needs and device‑dependent observations. It includes 400 videos—290 that require assistance with 555 intervention events, and 110 that should trigger silence—to test models’ ability to provide timely help and avoid unnecessary responses. Evaluations of twelve mainstream models show that only 4.48% of assistance videos are handled correctly, highlighting significant gaps in current systems.

By Mengze Hong, Ziwei Shang, Di Liang, Zijian Li, Liang Zhu, Hongyu Shan, Robin Zhou, Minlong Peng, Zhiqiang Kou, Jia Wang, Di Jiang, Chen Jason Zhang
benchmarks
More like this →
arXiv Computation and Language
Oct 22

MemoProAgent: Toward Memory-Driven Proactive Assistance

MemoProAgent tackles memory‑driven proactive assistance by mining long‑horizon personal memories to decide when an assistant should intervene. The authors introduce MemoProBench, a benchmark pairing egocentric video with year‑long, distractor‑rich user memories and human‑verified trigger windows, and show that existing models perform poorly (≤15%). MemoProAgent uses a graph‑organized memory store and a lightweight RL stage to improve accuracy from <10% to 42.4% without training.

By Shenglai Zeng, Xiao Yang, Tiffany Ma, Xinyuan Zhang, Nanshu Wang, Deren Lei, Kai Sun, Xilun Chen, Tammy Stark, Xianhui Zhu, Yuetai Li, Rulin Shao, Tianxin Wei, Rakesh Wanga, Anuj Kumar, Jiliang Tang, Wen-tau Yih, Xin Luna Dong
llmsragreinforcement-learningbenchmarks
More like this →
arXiv Computation and Language
Oct 19

ProPerAgent: Toward Memory-Driven Proactive Assistance

The paper introduces ProPerMem, a benchmark for memory-driven proactive assistance that pairs egocentric video with long, distractor-rich personal memories and human-verified trigger windows. Baseline models perform poorly, achieving at most 15% accuracy, while an oracle-driven GPT-5.3-Chat model reaches 96% when given evidence inside the trigger window. Building on these insights, the authors develop ProPerAgent, a plug‑and‑play system that retrieves relevant memory from a graph store and uses lightweight reinforcement learning to improve proactive intervention accuracy to 42.4% without training.

By Shenglai Zeng, Xiao Yang, Tiffany Ma, Xinyuan Zhang, Nanshu Wang, Deren Lei, Kai Sun, Xilun Chen, Tammy Stark, Xianhui Zhu, Yuetai Li, Rulin Shao, Tianxin Wei, Rakesh Wanga, Anuj Kumar, Jiliang Tang, Wen-tau Yih, Xin Luna Dong
llmsragagentsreinforcement-learningbenchmarks
More like this →
arXiv Machine Learning
Sep 24

Live Assistant: Learning Whether, When, and Whom to Assist in Real-World Live Social Streams

arXiv:2609. 27303v1 Announce Type: new Abstract: Livestreams are long-lasting interactive environments where audiovisual content, viewer activity, host behavior, and platform signals evolve together, creating assistance needs that emerge from the stream itself.

By Shujian Gao, Jiamei Yan, Yuchen Yang, Penghao Zhou, Qinglei Wang, Tiehan Fan, Yuan Wang, Zuxuan Wu, Yu-gang Jiang
fine-tuningmultimodalbenchmarks
More like this →