← Back to all news
arXiv Computer Vision By Chaofang Ma, Lin Jiang, Carol Jingyi Li, Xingyu Liu, Zeyu Li, Jiang Xu, Wei Zhang

Not All Attention Heads Contribute to Critical Visual Token Selection: Head-Aware Pruning Matters More

Read the original on arXiv Computer Vision →

The paper shows that only a small subset of attention heads in vision-language models is responsible for selecting critical visual tokens. By pruning tokens based on similarity before LLM reasoning and then applying head‑aware pruning during reasoning, the proposed ProViP framework achieves high task performance with significant speedups. Experiments on LLaVA‑1.5‑7B demonstrate 95.9% performance retention and a 1.62× inference speedup at an 88.9% pruning ratio.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computer Vision.

Related stories

arXiv Computer Vision
4d ago

LearnPruner: Rethinking Attention-based Token Pruning in Vision Language Models

LearnPruner rethinks token pruning in vision‑language models by addressing shortcomings of attention mechanisms in vision encoders and large language models. The method uses a two‑stage approach: first a learnable pruning module removes redundant vision tokens after the vision encoder, then only task‑relevant tokens are kept in the LLM’s middle layer. Experiments show that LearnPruner retains about 95% of performance while using only 5.5% of vision tokens, achieving a 3.2× speed‑up.

By Rinyoichi Takezoe, Yaqian Li, Zihao Bo, Anzhou Hou, Mo Guang, Kaiwen Long
llmsefficiencymultimodalsafety
More like this →
arXiv AI
Sep 10

STAR-Pro: Stage-Wise Token Adaptive Reduction with Progressive Refinement for Efficient Large Vision-Language Models

arXiv:2609.05916v1 Announce Type: cross Abstract: Large vision-language models (LVLMs) achieve strong multimodal understanding, but the hundreds to thousands of visual tokens they process impose subs...

By Yichen Guo, Tinghao Wang, Qizhe Zhang, Lingbei Meng, Yuan Zhang, Jiajun Cao, Hao Jiang, Chenwei Wu, Jixian Wu, Sixiang Chen, Tao Luo, Hongyang Cheng, Kai Tang, Chenxi Li, Renyuan Li, Xiande Huang, Wenya Wang, Shanghang Zhang
llmsefficiencymultimodalbenchmarks
More like this →
Hugging Face Trending Papers
Sep 28

When Text Matters: Design Principles for Visual Token Pruning in Vision-Language Model

The paper introduces a training‑free visual token pruning strategy for vision‑language models that separates early vision‑guided pruning from later text‑guided reselection. By first pruning tokens with vision‑encoder attention, retaining candidates until the decoder midpoint, and then applying text‑to‑visual attention, the method preserves task‑relevant visual information. Across eight benchmarks and three models, it achieves an average performance recovery of 11.10 and 16.84 percentage points at 80% and 90% pruning, respectively, while maintaining comparable or lower LLM‑prefill latency.

llmsefficiencymultimodalbenchmarks
More like this →