← Back to all news
arXiv Computation and Language By Jitai Hao, Ke Yang, Qiang Huang, Jun Yu

ShallowStream: Index Shallow then Answer Deep for Streaming Video Understanding

Read the original on arXiv Computation and Language →

ShallowStream is a framework for streaming video understanding that uses the shallow layers of a multimodal large language model (MLLM) to encode frames and build a lightweight index. During streaming, it maintains an always‑on index via the KV cache of shallow layers, and at query time it scores context frames using shallow‑layer attention and selects diverse evidence for answering. The approach matches the performance of leading streaming methods while cutting per‑frame prefill latency and 10‑second end‑to‑end latency by up to 52.1× and 11.9×, respectively.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computation and Language.

Related stories

arXiv Computer Vision
Sep 4

CoFiE: Coarse-to-Fine Evidence Selection for Efficient Streaming Video Understanding

CoFiE introduces a two‑stage evidence selection framework for streaming video understanding, separating a coarse, query‑agnostic filtering of visually distinctive frames from a fine, query‑specific refinement during LLM prefill. By filtering out redundant frames before expensive vision encoding, CoFiE reduces end‑to‑end latency while maintaining high accuracy. The method achieves state‑of‑the‑art performance on benchmarks such as StreamingBench and OvO‑Bench, improving accuracy by up to 3.15% and inference speed by up to 2.54× compared to prior approaches.

By Jing Jiang, Yiran Ling, Ruonan Li, Dimitrios Stamoulis, Jie Liu
llmsefficiencymultimodalbenchmarks
More like this →
arXiv AI
Jul 29

Reasoning with Memory: A Temporal Granularity-Adaptive Framework for Training-Free Long Video Understanding

arXiv:2607. 24794v1 Announce Type: new Abstract: While Multimodal Large Language Models (MLLMs) demonstrate superior generalization in fundamental video tasks, restricted context windows limit their long video understanding.

By Linghao Meng, Qiankun Li, Junyuan Mao, Pujin Liao, Zhicheng He, Enbo Zhang, Kun Wang, Yang Liu, Huazhu Fu, Yueming Jin
llmsmultimodalbenchmarkssafety
More like this →
arXiv AI
3d ago

Memory as Weights: Internalizing Long-Term History for Streaming Videos

The paper introduces MaLoW, a streaming memory framework that internalizes long‑term video history into evolving lightweight parameter updates using a recurrent HyperNetwork and LoRA weights. MaLoW keeps current observations as input while carrying historical context forward without re‑visiting the full video stream. The authors also present MaLoW‑Instruct, a large streaming video dataset, and show that MaLoW outperforms prior memory methods on several benchmarks and improves action success in a vision‑language‑action setting.

By Zhuoming Liu, Jinhong Lin, Lin Zhang, Yiwu Zhong, Yin Li
llmsfine-tuningmultimodal
More like this →