arXiv AI
Aug 5

Adaptive Two-Stage Visual Token Pruning for Efficient Inference in Video-Language Models

arXiv:2608. 03112v1 Announce Type: cross Abstract: Vision-language models excel at image and video understanding but suffer from high inference latency due to the need to process thousands of tokens per image, limiting their deployment on resource-constrained edge devices and in real-time surveillance applications.

By Paribesh Regmi, Qingshuang Chen, Chi Zhang, Heba Aly, Yelin Kim, Hongda Mao
arXiv AI
3d ago

VisCache: Visual KV Cache Pruning for Efficient Vision Large Language Model Inference

VisCache introduces a two-stage, plug‑and‑play framework for pruning visual key‑value caches in Vision Large Language Models without retraining. The first stage filters out temporally redundant keyframes, while the second stage, PruneKV, applies a parabolic layer‑wise budget and asymmetric update to selectively prune keys and fuse values, preserving essential context. Experiments show up to 2.35× speedup and significant memory savings with only 19–28% of the original cache retained, outperforming existing baselines.

By Lyuke Wang, Zhuo Li, Guangxu Zhu
arXiv Machine Learning
Jun 9

Beyond FLOPs: Benchmarking Real Inference Acceleration of LLM Pruning under a GEMM-Centric Taxonomy

arXiv:2606. 09080v1 Announce Type: new Abstract: Pruning has emerged as a dominant paradigm for accelerating large language model (LLM) inference, spanning a broad spectrum of methods that remove computation across tokens, layers, heads, dimensions, and attention patterns.

By Haozhe Hu, Hao Wu, Anhao Zhao, Longwei Ding, Peiran Yin, Yunpu Ma, Xiaoyu Shen