arXiv AI
Jun 2
AdaCodec: A Predictive Visual Code for Video MLLMs
arXiv:2606. 02569v1 Announce Type: cross Abstract: Video is temporally redundant: adjacent frames usually share most objects, background, and layout.
CodecSight: Leveraging Video Codec Signals for Efficient Streaming VLM Inference
Related stories
arXiv Computer Vision
Oct 1
RESUME: Recurrent State Updates from Motion and Residual Signals for Efficient Video Language Modeling
arXiv:2609.39563v1 Announce Type: new Abstract: Existing video language models encode sampled RGB frames independently, so a long video must either exhaust the token budget or drop the changes betwee...
arXiv Computer Vision
Aug 31
Visual Token Coding for Video Multimodal Large Language Models
The paper introduces Visual Token Coding (VTC), a token compression method for video multimodal large language models that mimics classical video coding by predicting I/P frames and measuring residuals to reduce token redundancy. VTC is extended with dynamic features—Dynamic Resolution Input, Dynamic Token Allocation, and Spatial Coverage Top‑K—forming VTC_Dy, which can be applied to existing MLLMs without additional tuning. Experiments on three MLLMs and multiple video benchmarks show that VTC_Dy retains over 100% of average performance with a 50% token budget and 97.8% with a 25% budget, while the code is publicly available.
arXiv AI
Jun 2
StreamingVLM: Real-Time Understanding for Infinite Video Streams
arXiv:2510. 09608v2 Announce Type: replace-cross Abstract: Vision-language models (VLMs) could power real-time assistants and autonomous agents, but they face a critical challenge: understanding near-infinite video streams without escalating latency and memory usage.
arXiv Machine Learning
Oct 22
Cache Locally, Recompute Selectively for Streaming Video Understanding
The paper introduces FrameBlend, a plug‑and‑play technique for streaming video understanding that selectively recomputes key‑value (KV) caches at query time. By re‑prefilling only query‑relevant video tokens, FrameBlend allows shorter attention windows during idle prefill, cutting prefill cost while boosting accuracy. Experiments on three SVU frameworks and five benchmarks show up to 3.9‑point accuracy gains and significant reductions in FLOPs and processing time.
arXiv Computer Vision
Aug 31
Token-Budget Distillation: Transferring Full-Token Semantics to Compressed Video Vision-Language Models
Token-Budget Distillation (TBD) is a parameter‑efficient fine‑tuning framework that adapts video vision‑language models to a fixed token budget. It freezes the pretrained backbone, updates only LoRA adapters, and incorporates FlashVID visual token compression. TBD uses a dual‑path teacher‑student design with full‑token supervision and compressed student optimization, enabling the student to recover full‑token semantics while remaining efficient under aggressive token reduction.
arXiv AI
Oct 15
HIMMEL: Hierarchical Interleaved Multi-stream Motion Encoding for Long Video Understanding
arXiv:2605.08158v2 Announce Type: replace-cross Abstract: Long video understanding with multimodal language models suffers from three compounding bottlenecks: heavy decode cost to obtain dense RGB fr...
arXiv Computer Vision
Sep 22
PREM: Prefix-Steered Recurrent Memory for Long-Video Understanding
arXiv:2609.23601v1 Announce Type: new Abstract: Long-video understanding must capture transient visual evidence under strict token budgets, yet existing methods compress frames, append memory tokens,...
arXiv AI
Jun 11
AVIS: Adaptive Test-Time Scaling for Vision-Language Models
arXiv:2606. 11576v1 Announce Type: cross Abstract: Modern Vision-Language Models (VLMs) benefit from chain-of-thought prompting and test-time scaling, but these gains often come with prohibitive inference cost due to large visual contexts and long decoding chains.
arXiv Computation and Language
Sep 10
Why Is Video Still So Expensive? A Survey of Inference-Efficiency Mechanisms in Video and Audiovisual LLMs
arXiv:2609.10355v1 Announce Type: cross Abstract: Video understanding has rapidly evolved toward video large language models (VideoLLMs): systems that couple video representations with pretrained lar...
arXiv Computer Vision
Sep 30
ReCaVSR: One-Step Streaming Diffusion Video Super-Resolution with Recycled Latents and Learned Cache Routing
arXiv:2609.37831v1 Announce Type: new Abstract: Real-time diffusion-based video super-resolution (VSR) is in high demand for online streaming, yet stringent latency requirements often compromise gene...
arXiv AI
Oct 5
Foresight: planning future perception in streaming VLMs without retraining
arXiv:2610.03123v1 Announce Type: cross Abstract: Existing streaming vision-language models (VLMs) continuously perceive and reason over visual streams, but their computational pathways remain fixed...