arXiv AI

Watch-Think-Interact: Bootstrapping Long-Horizon Multi-Turn Streaming Video Reasoning with Reinforcement Learning

The paper introduces Watch-Think-Interact (WTI), a closed-loop framework for multi-question streaming video reasoning that maintains compact natural-language memory entries linked to video time ranges. WTI decides whether to answer, continue watching, or recall relevant past intervals for each question, avoiding replay of the full history. The authors build a large dataset, WTI-82K, and a training method, Stream-GDPO, achieving state‑of‑the‑art performance on StreamingBench and OVO-Bench.

arXiv AI
3d ago

MemInStream: Memorizing or Merely Recognizing? Rethinking Streaming Video Memory Evaluation

MemInStream is a new benchmark designed to evaluate memory capabilities in streaming video understanding. It introduces an evidence‑controlled validation protocol that links each question to temporally localized evidence intervals, ensuring that models must rely on historical memory rather than recent observations or linguistic priors. The benchmark covers four key categories—Episodic Retrieval, State Tracking & Updating, Multi‑Evidence Integration, and Contextual Memory—across diverse video scenarios to systematically assess memory performance.

By Zhenqi He, Dunyuan Xu, Yang Bo, Kien T. Pham, Yanghao Wang, Shuhan Li, Xintong Guo, Yuyang He, Fang Liu, Huankang Guan, Jingyan Chen, Rui Liu, Long Chen
arXiv Computer Vision
Sep 2

StreamScout: Learning When to Look Deeper for Streaming Video Understanding

arXiv:2609.00291v1 Announce Type: new Abstract: Streaming video understanding requires answering questions that arrive at arbitrary moments over an unbounded video stream. Existing systems primarily...

By Ce Zhang, Jing Bi, Jinxi He, Jianshu Zhang, Jingyang Lin, Yunzhong Xiao, Minghao Fu, Yaqi Xie, Zhentao Xie, Weicong Chen, Katia Sycara, Ming Zhou
arXiv AI
Jun 17

LiveStarPro: Proactive Streaming Video Understanding with Hierarchical Memory for Long-Horizon Streams

arXiv:2606. 17798v1 Announce Type: cross Abstract: Despite the remarkable progress of Video Large Language Models (Video-LLMs), current online architectures still struggle to simultaneously process continuous video streams, decide autonomously when to respond, and preserve long-horizon contextual memory.

By Zhenyu Yang, Kairui Zhang, Bing Wang, Shengsheng Qian, Changsheng Xu
arXiv AI
Jun 24

video-SALMONN-R$^3$: Learning to ReWatch, ReAsk, and ReAnswer for Efficient Video Understanding

arXiv:2606. 24477v1 Announce Type: cross Abstract: Video large language models (LLMs) are often constrained by computation and memory budgets, leading them to use reduced frame rates and spatial resolutions, which may cause them to miss critical information for question answering (QA).

By Yixuan Li, Guangzhi Sun, Yudong Yang, Wei Li, Zejun MA, Chao Zhang
arXiv AI
Oct 16

Memory as Weights: Internalizing Long-Term History for Streaming Videos

The paper introduces MaLoW, a streaming memory framework that internalizes long‑term video history into evolving lightweight parameter updates using a recurrent HyperNetwork and LoRA weights. MaLoW keeps current observations as input while carrying historical context forward without re‑visiting the full video stream. The authors also present MaLoW‑Instruct, a large streaming video dataset, and show that MaLoW outperforms prior memory methods on several benchmarks and improves action success in a vision‑language‑action setting.

By Zhuoming Liu, Jinhong Lin, Lin Zhang, Yiwu Zhong, Yin Li
arXiv Computation and Language
Oct 26

SyncPro: Synchronous Reasoning and Responding for Proactive Streaming Video Understanding

arXiv:2610.32132v1 Announce Type: cross Abstract: Proactive streaming video understanding requires assistants to track unfolding events and respond when a user's requested condition is met. To make t...

By Xueyun Tian, Bingbing Xu, Jiaming Liu, Jinpeng Yu, Yang Shi, Yu Zhao, Ruihua Huang, Yuanzhuo Wang, Huawei Shen
arXiv Computer Vision
Nov 2

TwinPast: Benchmarking Historical Evidence Use in Streaming Video Understanding with Evidence Contrastive Video Pairs

TwinPast is a new benchmark that tests whether streaming video models use historical evidence by presenting pairs of videos that differ only in critical past events while keeping recent context, questions, and options the same. The study evaluates 34 model configurations on 350 human‑reviewed pairs across five historical tasks, finding that most models rely on recent frames or the question alone, with only a few showing improved performance when historical inputs are provided. Even the best streaming system scores only 49.14% PairAcc, far below general‑purpose models, and case studies reveal that models often select states from the wrong time or contradict the recorded event order.

By Haotian Yang, Yong Chen, Cheng Peng, Haobo Wang, Zhongle Xie, Xuanru Li, Gang Chen
Hugging Face Trending Papers
Sep 3

Beyond Retrieval: Progressive Latent Memory Evolution for Streaming Video Understanding

The paper introduces LatentStream, a progressive latent working memory framework for streaming video understanding. It replaces the traditional store‑and‑retrieve approach with a retrieve‑and‑internalize strategy, organizing visual history into short, mid, and long‑term levels and progressively expanding memory receptive fields to internalize evidence into a compact latent memory. The method also employs confidence‑guided optimization to refine memory tokens, achieving state‑of‑the‑art results on online and offline video benchmarks.

arXiv Computer Vision
Oct 2

OneStreamer: Unifying Perception, Memory, and Proactive Response in Streaming Video Interaction

OneStreamer is a streaming video model that jointly learns to record evidence and respond to tasks through a shared proactive generation process. Its Proactive Hierarchical Caption Memory creates time‑grounded local‑detail captions and event summaries, while Proactive State Transition Learning reduces waiting states by supervising all output anchors. The authors also built a large OneStreamer‑1M dataset and show that a 4B model outperforms baselines on eight streaming video benchmarks, with ablations confirming the benefits of generated captions and PSTL.

By Xiangyu Zeng, Yuandong Yang, Zhiqiu Zhang, Yuhan Zhu, Xinhao Li, Qingyi Si, Dingyu Yao, Changlian Ma, Haoran Chen, Xinyu Chen, Yansong Shi, Junhao Zhou, Yifei Li, Jun Zhang, Chuanyu Qin, Chenxu Yang, Xinlei Yu, Kun Ouyang, Yuchen Shao, Qianshan Wei, Changhai Zhou, Jun Gao, Jiaqi Wang, Limin Wang