SinkPruner is a training‑free framework that prunes visual tokens for multimodal large language models by first removing high‑norm redundant tokens with a visual sanitizer and then selectively keeping tokens that align with the text query using a text‑guided pruner. The coarse‑to‑fine design reduces attention sink and dispersion, enabling an 89% token reduction while preserving 96.5% of LLaVA‑1.5’s performance and 91.8% of Qwen2.5‑VL’s performance across twelve image‑language and four video‑language benchmarks. The visual sanitizer also improves existing pruning methods, showing strong transferability.
By Shiyu Li, Zi-Yuan Hu, Shijia Huang, Yanyang Li, Yiwu Zhong, Liwei Wang
arXiv:2609.05916v1 Announce Type: cross
Abstract: Large vision-language models (LVLMs) achieve strong multimodal understanding, but the hundreds to thousands of visual tokens they process impose subs...
By Yichen Guo, Tinghao Wang, Qizhe Zhang, Lingbei Meng, Yuan Zhang, Jiajun Cao, Hao Jiang, Chenwei Wu, Jixian Wu, Sixiang Chen, Tao Luo, Hongyang Cheng, Kai Tang, Chenxi Li, Renyuan Li, Xiande Huang, Wenya Wang, Shanghang Zhang
arXiv:2609.36916v1 Announce Type: new
Abstract: Multimodal large language models (MLLMs) incur high inference latency from long visual token sequences. Existing pruning methods commonly use attention...
By Weixuan Li, Zikun Zhou, Xinyi Zhuang, Xinyan Guo, Rui Tian, Chuyao Zhang, Lin Gao
arXiv:2608. 07193v1 Announce Type: new Abstract: Visual-token pruning can substantially reduce the inference cost of multimodal large language models (MLLMs), yet existing methods largely rely on fixed, handcrafted heuristics and costly expert trial and error.
By Zhen Liu, Wenli Huang, Wei Song, Yuhan Liu, Zhiqin Yang, Jingwen Fu
arXiv:2609.34330v2 Announce Type: replace
Abstract: Multimodal large language models (MLLMs) have demonstrated impressive performance in multimodal understanding, but processing large numbers of visu...
By Tinghao Wang, Yichen Guo, Qizhe Zhang, Yuan Zhang, Weimin Ouyang, Rui Huang, Jiajun Cao, Sixiang Chen, Hao Jiang, Jixian Wu, Zheng Lu, Bofan Zhu, Renyuan Li, Shanghang Zhang
The paper shows that only a small subset of attention heads in vision-language models is responsible for selecting critical visual tokens. By pruning tokens based on similarity before LLM reasoning and then applying head‑aware pruning during reasoning, the proposed ProViP framework achieves high task performance with significant speedups. Experiments on LLaVA‑1.5‑7B demonstrate 95.9% performance retention and a 1.62× inference speedup at an 88.9% pruning ratio.
By Chaofang Ma, Lin Jiang, Carol Jingyi Li, Xingyu Liu, Zeyu Li, Jiang Xu, Wei Zhang
arXiv:2606. 27161v1 Announce Type: new Abstract: Multimodal large language models (MLLMs) have achieved strong multimodal reasoning capabilities, but their efficiency is limited by the large number of visual tokens, which introduces substantial computational overhead.
By Tinghao Wang, Yichen Guo, Rui Huang, Zheng Lu, Qizhe Zhang, Chenxi Li, Yuan Zhang, Jiajun Cao, Zhirong Shen, Yaosong Du, Guangyan Gan, Wenya Wang, Lin William Cong, Shanghang Zhang
arXiv:2603. 00198v2 Announce Type: replace-cross Abstract: Token reduction accelerates long-video vision--language models (VLMs), but existing methods target Transformers, where reduction is treated as token pruning.
By Jindong Jiang, Amala Sanjay Deshmukh, Kateryna Chumachenko, Karan Sapra, Zhiding Yu, Guilin Liu, Andrew Tao, Pavlo Molchanov, Jan Kautz, Wonmin Byeon
arXiv:2609.13804v1 Announce Type: new
Abstract: Visual prefixes account for a major portion of the per-layer computation in multimodal large language models (MLLMs), making visual-token pruning a dir...
By Hansen Zhang, Landi He, Mingde Yao, Lijian Xu
arXiv:2607. 07033v1 Announce Type: cross Abstract: Large vision-language models incur substantial inference costs because high-resolution inputs introduce thousands of visual tokens, many of which are redundant for a given query.
By Kyuan Oh, Bumsoo Kim
The paper introduces Adaptive Visual Token Pruning (AVTP), a training‑free framework that dynamically selects pruning layers and ratios for large vision‑language models (LVLMs) when processing multiple image sequences. By analyzing visual attention distributions across different LVLM architectures, AVTP adapts token retention to image importance, enabling efficient inference without relying on attention‑based computations incompatible with FlashAttention. Experiments show significant speedups—up to 2× for Qwen3VL‑8B—while preserving or even improving accuracy on multi‑image benchmarks.
By Rongyang Zhang, Chengqiang Lu, Cong Li, Hongchao Gu, Tingjia Shen, Xuyang Zhi, Qimeng Wang, Yan Gao, Yi Wu, Yao Hu, Hao Wang, Enhong Chen
The paper introduces Role-Conditioned Sub-Token Routing (RoleSub), a method that compresses the value representations of retained tokens in Vision‑Language‑Action models. By partitioning each token’s value into orthogonal groups and routing them based on token features, a latent role, and language context, RoleSub can also compress language values. Experiments on OpenVLA‑OFT‑7B show that, at matched visual‑KV budgets, RoleSub outperforms token‑only control in most settings and reduces total KV to 9.2–11.3% of the original while maintaining strong control performance.
By Wei Jiang, Wei Wang