arXiv Computer Vision

Visual Token Coding for Video Multimodal Large Language Models

The paper introduces Visual Token Coding (VTC), a token compression method for video multimodal large language models that mimics classical video coding by predicting I/P frames and measuring residuals to reduce token redundancy. VTC is extended with dynamic features—Dynamic Resolution Input, Dynamic Token Allocation, and Spatial Coverage Top‑K—forming VTC_Dy, which can be applied to existing MLLMs without additional tuning. Experiments on three MLLMs and multiple video benchmarks show that VTC_Dy retains over 100% of average performance with a 50% token budget and 97.8% with a 25% budget, while the code is publicly available.

arXiv AI

Jun 2

AdaCodec: A Predictive Visual Code for Video MLLMs

arXiv:2606. 02569v1 Announce Type: cross Abstract: Video is temporally redundant: adjacent frames usually share most objects, background, and layout.

By Haowen Hou, Zhen Huang, Zheming Liang, Qingyi Si, Chenglin Li, Shuai Dong, Kele Shao, Ruilin Li, Dianyi Wang, Nan Duan, Jiaqi Wang

llms multimodal benchmarks

arXiv Computer Vision

4d ago

Token-Budget Distillation: Transferring Full-Token Semantics to Compressed Video Vision-Language Models

Token-Budget Distillation (TBD) is a parameter‑efficient fine‑tuning framework that adapts video vision‑language models to a fixed token budget. It freezes the pretrained backbone, updates only LoRA adapters, and incorporates FlashVID visual token compression. TBD uses a dual‑path teacher‑student design with full‑token supervision and compressed student optimization, enabling the student to recover full‑token semantics while remaining efficient under aggressive token reduction.

By Xiaoyang Guo, Guoping Luo, Jusheng Zhang, Keze Wang, Wenhao Wang

llms fine-tuning efficiency multimodal benchmarks

Hugging Face Trending Papers

Aug 3

CRAFT: Compression via Recursive Adaptive Fusion of Video Tokens for Vision-Language Models

In video understanding, vision-language models (VLMs) must ingest massive numbers of visual tokens, causing the computational and memory cost of the prefill stage to rise sharply. Such visual sequences are highly redundant along the spatio-temporal dimension, yet a high compression ratio is often accompanied by the loss of critical details.

llms multimodal benchmarks safety

arXiv Computer Vision

Aug 24

Aggregating Visual Information with Optimal Transport for VideoLM Token Compression

arXiv:2608.20473v1 Announce Type: new Abstract: Video language models process videos as dense visual-token sequences with substantial representational redundancy. Compressing these sequences is there...

By Wenti Yin, Xiaotian Han, Junyuan Shang, Yuchen Ding, Shuohuan Wang, Dianhai Yu, Changxin Gao, Nong Sang

llms benchmarks

arXiv AI

Aug 12

Hybrid Token Compression for Vision-Language Models

arXiv:2512. 08240v2 Announce Type: replace-cross Abstract: Vision-language models (VLMs) rely on hundreds of visual tokens, leading to high computational and memory costs.

By Jusheng Zhang, Xiaoyang Guo, Tongyu Mo, Qinhan Lv, Wenhao Chai, Jian Wang, Keze Wang, Liang Lin

llms efficiency multimodal benchmarks

arXiv Computer Vision

Aug 25

Adaptive Tokenisation Via Temporal Redundancy Masking And Latent Inpainting

arXiv:2606.06158v2 Announce Type: replace Abstract: Adaptive video tokenisation seeks to dynamically allocate token budgets based on the underlying visual complexity of a sequence. Current continuous...

By Kevin Dave, Sai Aditya Patkuri, Chhaya Kumar Das, Gouranga Bala, Rajeshkumar SA, R. Venkatesh Babu

llms diffusion benchmarks

arXiv Machine Learning

Jun 11

Visual-Redundancy-Controlled Parallel Decoding for Diffusion-Based Multimodal Large Language Models

arXiv:2605. 25820v2 Announce Type: replace Abstract: Diffusion-based multimodal large language models (dMLLMs) decode by iteratively predicting tokens at multiple masked positions in parallel.

By Yulin Yuan, Hongshuo Zhao, Xiangming Meng

llms diffusion multimodal benchmarks

arXiv AI

Jul 2

Stateful Token Reduction for Long-Video Hybrid VLMs

arXiv:2603. 00198v2 Announce Type: replace-cross Abstract: Token reduction accelerates long-video vision--language models (VLMs), but existing methods target Transformers, where reduction is treated as token pruning.

By Jindong Jiang, Amala Sanjay Deshmukh, Kateryna Chumachenko, Karan Sapra, Zhiding Yu, Guilin Liu, Andrew Tao, Pavlo Molchanov, Jan Kautz, Wonmin Byeon

llms fine-tuning efficiency benchmarks

arXiv AI

Aug 5

Adaptive Two-Stage Visual Token Pruning for Efficient Inference in Video-Language Models

arXiv:2608. 03112v1 Announce Type: cross Abstract: Vision-language models excel at image and video understanding but suffer from high inference latency due to the need to process thousands of tokens per image, limiting their deployment on resource-constrained edge devices and in real-time surveillance applications.

By Paribesh Regmi, Qingshuang Chen, Chi Zhang, Heba Aly, Yelin Kim, Hongda Mao

llms rag fine-tuning efficiency multimodal benchmarks

arXiv AI

Jun 9

OmniMem: Perturbation-aware Memory Compression for Streaming Audio-Visual LLMs

arXiv:2606. 07577v1 Announce Type: new Abstract: Audio-visual large language models (LLMs) hold strong promise for long-form video understanding, yet their long-video inference is fundamentally limited by the linear growth of video tokens and key-value (KV) caches.

By Guangzhi Sun, Yixuan Li, Yudong Yang, Chao Zhang

llms fine-tuning multimodal

arXiv AI

Aug 28

Position Is All You Need: A Free Lunch Token Compression Strategy for MLLM-based Referring Expression Segmentation

The paper introduces PAYN, a training‑free token compression strategy for Multimodal Large Language Model (MLLM) based Referring Expression Segmentation (RES). By preserving original position embeddings and local spatial structures, PAYN retains tokens that are evenly distributed across neighboring regions, thereby maintaining spatial relational consistency. Experiments on multiple RES benchmarks show that PAYN outperforms existing token compression methods, confirming that position information alone is sufficient for effective compression in this task.

By Yuhan Liu, Yixiong Zou, Yuhua Li, Ruixuan Li

llms rag computer-vision multimodal benchmarks

arXiv AI

Jul 13

M4V: Multimodal Mamba for Efficient Text-to-Video Generation

arXiv:2506. 10915v2 Announce Type: replace-cross Abstract: Text-to-video generation has significantly enriched content creation and holds the potential to evolve into powerful world simulators.

By Jiancheng Huang, Gengwei Zhang, Zequn Jie, Siyu Jiao, Yinlong Qian, Ling Chen, Yunchao Wei, Lin Ma

llms diffusion multimodal benchmarks

Related stories