SVMemAgent introduces a streaming video memory (SVMem) that continuously updates a compact representation of observed frames for online keyframe selection without prior knowledge of video length, query, or future frames. The agent decides at each timestep whether to replace an existing memory frame with a new one or discard it, trained via Group Relative Policy Optimization using task-driven rewards from question-answer pairs. Experiments demonstrate that SVMemAgent outperforms existing online baselines and rivals offline methods, and its learned policy tends to favor frames containing textual information, potentially aiding downstream VideoQA tasks.
Most keyframe selection studies focus on offline settings, assuming access to the full video and query in advance. In contrast, real-world streaming scenarios require online frame selection under unkn...
arXiv:2606. 16353v1 Announce Type: cross Abstract: Streaming video understanding models must answer queries at any moment during an ongoing stream, using only what they have observed so far and under fixed memory and computation budgets.
arXiv:2609.00291v1 Announce Type: new Abstract: Streaming video understanding requires answering questions that arrive at arbitrary moments over an unbounded video stream. Existing systems primarily...
The paper introduces LatentStream, a progressive latent working memory framework for streaming video understanding. It replaces the traditional store‑and‑retrieve approach with a retrieve‑and‑internalize strategy, organizing visual history into short, mid, and long‑term levels and progressively expanding memory receptive fields to internalize evidence into a compact latent memory. The method also employs confidence‑guided optimization to refine memory tokens, achieving state‑of‑the‑art results on online and offline video benchmarks.
The paper introduces LatentStream, a progressive latent working memory framework for streaming video understanding that replaces the traditional store‑and‑retrieve paradigm with a retrieve‑and‑internalize approach. It organizes visual history into short, mid, and long‑term levels using Jenks‑guided adaptive consolidation, then expands memory receptive fields to iteratively retrieve and internalize evidence into a compact latent memory. A confidence‑guided optimization further refines this memory, leading to state‑of‑the‑art performance on online and offline video benchmarks.
The paper introduces Watch-Think-Interact (WTI), a closed-loop framework for multi-question streaming video reasoning that maintains compact natural-language memory entries linked to video time ranges. WTI decides whether to answer, continue watching, or recall relevant past intervals for each question, avoiding replay of the full history. The authors build a large dataset, WTI-82K, and a training method, Stream-GDPO, achieving state‑of‑the‑art performance on StreamingBench and OVO-Bench.
arXiv:2607. 24794v1 Announce Type: new Abstract: While Multimodal Large Language Models (MLLMs) demonstrate superior generalization in fundamental video tasks, restricted context windows limit their long video understanding.
The paper introduces Caption‑once, Frames‑on‑Demand (CFD), a budget‑aware edge‑cloud framework for long‑video understanding. CFD first runs a single offline captioning pass on the edge to build a dual‑track narrative index—an event‑level story skeleton and a clip‑level micro‑log—that is cached for future queries. At query time, a cloud‑side MLLM uses a Visual‑Need Router to decide whether to retrieve keyframes for perceptual questions, thereby limiting visual processing while preserving temporal structure in language space.
arXiv:2506. 01274v2 Announce Type: replace-cross Abstract: Recent progress in Large Multi-modal Models (LMMs) has enabled effective vision-language reasoning, yet the ability to video understanding remains constrained by suboptimal frame selection strategies, albeit with the rapid development of video-specialized LMMs.
arXiv:2602. 01801v2 Announce Type: replace-cross Abstract: Autoregressive video diffusion models enable streaming generation, opening the door to long-form synthesis, video world models, and interactive neural game engines.
The paper introduces AllocEmbed, an allocate‑then‑embed framework that reallocates a fixed visual‑input budget across more video frames to improve retrieval performance. A lightweight allocator uses low‑cost previews to assign frame‑wise resolutions before the embedding backbone, preserving detail where it most benefits retrieval while reducing visual cost elsewhere. Retrieval‑Driven Policy Optimization (RDPO) learns the allocator directly from retrieval feedback, and the method integrates with existing systems without modifying the embedding model.
arXiv:2608. 08612v1 Announce Type: cross Abstract: Recently, retrieval-augmented and memory-augmented methods have emerged as two promising paradigms for long-video question answering.