arXiv AI
Jun 2
AdaCodec: A Predictive Visual Code for Video MLLMs
arXiv:2606. 02569v1 Announce Type: cross Abstract: Video is temporally redundant: adjacent frames usually share most objects, background, and layout.
CodecSight: Leveraging Video Codec Signals for Efficient Streaming VLM Inference
Related stories
arXiv Computer Vision
Aug 31
Visual Token Coding for Video Multimodal Large Language Models
The paper introduces Visual Token Coding (VTC), a token compression method for video multimodal large language models that mimics classical video coding by predicting I/P frames and measuring residuals to reduce token redundancy. VTC is extended with dynamic features—Dynamic Resolution Input, Dynamic Token Allocation, and Spatial Coverage Top‑K—forming VTC_Dy, which can be applied to existing MLLMs without additional tuning. Experiments on three MLLMs and multiple video benchmarks show that VTC_Dy retains over 100% of average performance with a 50% token budget and 97.8% with a 25% budget, while the code is publicly available.
arXiv AI
Jun 2
StreamingVLM: Real-Time Understanding for Infinite Video Streams
arXiv:2510. 09608v2 Announce Type: replace-cross Abstract: Vision-language models (VLMs) could power real-time assistants and autonomous agents, but they face a critical challenge: understanding near-infinite video streams without escalating latency and memory usage.
arXiv Computer Vision
Aug 31
Token-Budget Distillation: Transferring Full-Token Semantics to Compressed Video Vision-Language Models
Token-Budget Distillation (TBD) is a parameter‑efficient fine‑tuning framework that adapts video vision‑language models to a fixed token budget. It freezes the pretrained backbone, updates only LoRA adapters, and incorporates FlashVID visual token compression. TBD uses a dual‑path teacher‑student design with full‑token supervision and compressed student optimization, enabling the student to recover full‑token semantics while remaining efficient under aggressive token reduction.
arXiv AI
Jun 11
AVIS: Adaptive Test-Time Scaling for Vision-Language Models
arXiv:2606. 11576v1 Announce Type: cross Abstract: Modern Vision-Language Models (VLMs) benefit from chain-of-thought prompting and test-time scaling, but these gains often come with prohibitive inference cost due to large visual contexts and long decoding chains.
arXiv Computation and Language
Sep 10
Why Is Video Still So Expensive? A Survey of Inference-Efficiency Mechanisms in Video and Audiovisual LLMs
arXiv:2609.10355v1 Announce Type: cross Abstract: Video understanding has rapidly evolved toward video large language models (VideoLLMs): systems that couple video representations with pretrained lar...
Hugging Face Trending Papers
Jul 21
Detect Early, Escalate Rarely: Anytime Detection of AI-Generated Video from the Compressed Bitstream
Detectors for AI-generated video are evaluated offline. A clip is decoded to pixels and scored once, increasingly by a large vision-language model.
arXiv Machine Learning
Jul 23
HeadCast: Casting Attention Heads for Efficient Autoregressive Video Generation
arXiv:2607. 20125v1 Announce Type: cross Abstract: Autoregressive (AR) video diffusion models have become a promising paradigm for long and streaming video synthesis, but the continuously growing Key-Value (KV) cache makes attention the dominant inference cost, especially at high resolution where each frame contributes many tokens.
arXiv Computation and Language
Sep 3
ShallowStream: Index Shallow then Answer Deep for Streaming Video Understanding
ShallowStream is a framework for streaming video understanding that uses the shallow layers of a multimodal large language model (MLLM) to encode frames and build a lightweight index. During streaming, it maintains an always‑on index via the KV cache of shallow layers, and at query time it scores context frames using shallow‑layer attention and selects diverse evidence for answering. The approach matches the performance of leading streaming methods while cutting per‑frame prefill latency and 10‑second end‑to‑end latency by up to 52.1× and 11.9×, respectively.
arXiv AI
Jun 9
OmniMem: Perturbation-aware Memory Compression for Streaming Audio-Visual LLMs
arXiv:2606. 07577v1 Announce Type: new Abstract: Audio-visual large language models (LLMs) hold strong promise for long-form video understanding, yet their long-video inference is fundamentally limited by the linear growth of video tokens and key-value (KV) caches.
arXiv AI
Jun 29
MLVC: Multi-platform Learned Video Codec for Real-World Deployment
arXiv:2606. 28027v1 Announce Type: cross Abstract: Neural video codecs have surpassed classical codecs in coding efficiency but remain impractical for deployment due to cross-platform incompatibility and high computational cost.
arXiv Computer Vision
Sep 4
CoFiE: Coarse-to-Fine Evidence Selection for Efficient Streaming Video Understanding
CoFiE introduces a two‑stage evidence selection framework for streaming video understanding, separating a coarse, query‑agnostic filtering of visually distinctive frames from a fine, query‑specific refinement during LLM prefill. By filtering out redundant frames before expensive vision encoding, CoFiE reduces end‑to‑end latency while maintaining high accuracy. The method achieves state‑of‑the‑art performance on benchmarks such as StreamingBench and OvO‑Bench, improving accuracy by up to 3.15% and inference speed by up to 2.54× compared to prior approaches.