CatchUp is a training‑free framework that improves efficiency in large vision‑language models by depth‑aware re‑admission of visual tokens. It repeatedly evaluates query‑conditioned relevance, processes only selected tokens per decoder interval, and preserves deferred tokens for later re‑admission, using reference‑to‑current transition signals to approximate missed representation evolution. Experiments on three referring‑expression grounding datasets and six multimodal benchmarks show that CatchUp retains over 92% of full‑model performance with 88.9% fewer active tokens and reduces end‑to‑end latency by 8.0% at a 64‑token budget.
By Eunju Park, Uiwon Hwang
arXiv:2606. 12412v1 Announce Type: cross Abstract: Vision-language models (VLMs) project images into hundreds to thousands of visual tokens, making decoder inference expensive in both attention computation and KV-cache memory.
By Cheng-Yu Yang, Shao-Yuan Lo, Yu-Lun Liu
LearnPrunerV2 is a fully learnable token‑pruning framework for Vision‑Language Models that replaces heuristic pruning with lightweight modules in every backbone layer. These modules estimate token importance by combining visual saliency and query relevance via a shadow token, revealing a consistent staircase redundancy pattern across layers. The method introduces Budget‑Deterministic Pruning to set fixed per‑layer budgets, achieving state‑of‑the‑art efficiency while preserving 99.8% of performance even after pruning 77.8% of visual tokens on LLaVA‑v1.5.
By Rinyoichi Takezoe, Yaqian Li, Weiren Wang, Zihao Bo, Huihong Shi, Anzhou Hou, Mo Guang, Jiale Yan, Kaiwen Long
LearnPrunerV2 introduces a fully learnable token pruning framework for Vision‑Language Models, embedding lightweight modules in every backbone layer to estimate token importance by combining visual saliency and query relevance via a shadow token. It identifies a consistent staircase redundancy pattern across layers and employs Budget‑Deterministic Pruning to set fixed per‑layer budgets, yielding a hardware‑friendly pruning schedule. Experiments show that LearnPrunerV2 outperforms prior methods, preserving 99.8% of performance while removing 77.8% of visual tokens on LLaVA‑v1.5, thus achieving a superior accuracy‑efficiency trade‑off.
arXiv:2606. 31903v1 Announce Type: cross Abstract: Multimodal large language models (MLLMs) increasingly process long visual-token sequences, increasing the overall inference computation.
By Zhaoyang Luo, Runmin Dong, Miao Yang, Fan Wei, Yushan Lai, Bin Luo, Haohuan Fu
arXiv:2606. 09131v1 Announce Type: new Abstract: Multimodal large language models (MLLMs) commonly inherit the deep, symmetric Transformer backbone designed for unimodal text modeling, and apply the same computation uniformly to image and language tokens.
By Siyuan Liu, Jinyang Wu
The paper introduces Role-Conditioned Sub-Token Routing (RoleSub), a method that compresses the value representations of retained tokens in Vision‑Language‑Action models. By partitioning each token’s value into orthogonal groups and routing them based on token features, a latent role, and language context, RoleSub can also compress language values. Experiments on OpenVLA‑OFT‑7B show that, at matched visual‑KV budgets, RoleSub outperforms token‑only control in most settings and reduces total KV to 9.2–11.3% of the original while maintaining strong control performance.
By Wei Jiang, Wei Wang
The paper introduces Role-Conditioned Sub-Token Routing (RoleSub), a method that compresses the value representations of retained tokens in Vision‑Language‑Action models. By partitioning each token’s value into orthogonal groups and routing them based on token features, latent roles, and language context, RoleSub can compress both visual and language representations without discarding tokens. Experiments on OpenVLA‑OFT‑7B show that, at matched visual‑KV budgets, RoleSub outperforms token‑only control in most settings and can reduce total KV to 9.2–11.3% of the original while maintaining strong control performance.
arXiv:2610.03389v1 Announce Type: cross
Abstract: Vision language models (VLMs) incur substantial inference cost because every visual token is processed by the attention and MLP projections of every...
By Rahul Chowdhury, Timothy A Rupprecht, Xuan Shen, Shaoyi Huang, Pu Zhao, Yanzhi Wang
arXiv:2608. 07193v1 Announce Type: new Abstract: Visual-token pruning can substantially reduce the inference cost of multimodal large language models (MLLMs), yet existing methods largely rely on fixed, handcrafted heuristics and costly expert trial and error.
By Zhen Liu, Wenli Huang, Wei Song, Yuhan Liu, Zhiqin Yang, Jingwen Fu
arXiv:2607. 26596v1 Announce Type: cross Abstract: Multimodal large language models (MLLMs) have demonstrated remarkable capabilities by integrating visual and textual understanding within a unified transformer architecture.
By Mingkuan Feng, Zhengqi Wen, Jianhua Tao
arXiv:2609.10346v1 Announce Type: new
Abstract: Multimodal large language models (MLLMs) process hundreds or thousands of visual tokens per image, incurring prohibitive inference costs. While existin...
By Haiji Liang, Pengfei Zhou, Zhenglin Wan, Wei Wang, Yang You, Wangbo Zhao