arXiv AI
Aug 12
Hybrid Token Compression for Vision-Language Models
arXiv:2512. 08240v2 Announce Type: replace-cross Abstract: Vision-language models (VLMs) rely on hundreds of visual tokens, leading to high computational and memory costs.
PACE: Pareto-Adaptive Compression for Efficient Native MLLMs
The paper introduces PACE, a Pareto‑adaptive compression framework designed for native multimodal large language models (MLLMs) that directly tokenize pixel patches without external vision encoders. By analyzing token discriminability and heterogeneous redundancy across spatial dimensions, PACE decomposes attention into height‑ and width‑specific saliency maps and applies a Pareto‑optimization strategy to allocate token budgets, achieving near‑lossless performance with only 25‑50% of visual tokens and more than 2× FLOP savings. Experiments demonstrate that PACE not only preserves but sometimes improves benchmark results compared to existing ViT‑based compression methods.
Related stories
arXiv Computer Vision
Sep 30
Representation Dynamics Reveal Semantic Saliency and Similarity for Visual Token Pruning in MLLMs
arXiv:2609.36916v1 Announce Type: new Abstract: Multimodal large language models (MLLMs) incur high inference latency from long visual token sequences. Existing pruning methods commonly use attention...
arXiv AI
Aug 28
PACE: A Unified Condense-and-Extract Paradigm for Fast VLM Inference
PACE introduces a training‑free Condense‑and‑Extract framework that speeds up Vision‑Language Model inference by first adaptively downsampling visual inputs before encoding and then selectively retaining essential tokens during decoding. The Adaptive Pixel Compressor (APC) reduces encoder workload while preserving global context, and the Dynamic Dual‑Attention Extractor (DDAE) keeps task‑critical details by fusing visual and language signals. Applied to Qwen2.5‑VL‑7B, PACE maintains 93.8% of performance using only 10% of visual tokens, achieving a 3.1× speedup in time to first token.
arXiv Computer Vision
Oct 27
Rescuing Your Pruning: Rethinking Visual Token Pruning and Post-Training Recovery in MLLMs
arXiv:2610.33688v1 Announce Type: new Abstract: Visual token pruning substantially reduces the inference cost of multimodal large language models (MLLMs), yet existing methods are mostly training-fre...
arXiv AI
Sep 10
STAR-Pro: Stage-Wise Token Adaptive Reduction with Progressive Refinement for Efficient Large Vision-Language Models
arXiv:2609.05916v1 Announce Type: cross Abstract: Large vision-language models (LVLMs) achieve strong multimodal understanding, but the hundreds to thousands of visual tokens they process impose subs...
Hugging Face Trending Papers
Aug 11
When Vision Becomes Text: Visual Token Pruning via Cross-Modal Residual Guidance in VLMs
Abundant visual information strengthens vision-language model (VLM) perception, yet massive visual tokens raise inference costs. Existing visual token pruning methods rely on similarity-based guidance, which exploits pairwise text-vision and vision-vision token correlations for compression.
arXiv AI
Jul 31
Decoupled Visual Processing: Efficient Multimodal Adaptation via Modality-Specific Transformer Substitution
arXiv:2607. 26596v1 Announce Type: cross Abstract: Multimodal large language models (MLLMs) have demonstrated remarkable capabilities by integrating visual and textual understanding within a unified transformer architecture.
arXiv AI
Sep 30
ResComEmb: Effective and Efficient Multimodal Embedding via Residual Homogeneity Compression
arXiv:2609.37225v1 Announce Type: cross Abstract: Multimodal large language models (MLLMs) have shown strong potential for universal multimodal representation learning. However, existing methods eith...
arXiv AI
Oct 2
MWOP: Modality-aware Width-wise Operation Pruning for Efficient MLLMs
MWOP (Modality-aware Width-wise Operation Pruning) is a method that independently prunes visual‑to‑visual, text‑to‑visual, and text‑to‑text attention paths within each layer of multimodal large language models, and separately selects feed‑forward network channels for visual and textual inputs. It uses a first‑order Taylor criterion to guide pruning, re‑evaluates FFN importance after attention pruning, and applies LoRA‑based recovery training. The approach is paired with path‑sparse Triton attention kernels and compact visual‑side FFN execution to achieve practical acceleration, preserving token sequences while reducing computation. "whyItMatters":"MWOP achieves a 1.6× prefill speedup on LLaVA‑OneVision‑7B while retaining 99.7% performance, and further boosts token‑compression methods to 2.9× and 2.7× speedups, demonstrating its effectiveness across architectures."
arXiv Machine Learning
Oct 9
V-CoLA: Vision Token Compression with Linear Attention
The paper introduces V-CoLA, a training‑free token compression framework tailored for vision-language models that use linear attention. V-CoLA employs a uniqueness‑aware importance criterion to select critical vision tokens and an adaptive token merging strategy to compress them, all optimized to fit the chunk‑wise parallelism of linear attention. Experiments show that V‑CoLA retains 99.5% of the original performance with only half the vision tokens and achieves significant speedups during prefill.
arXiv Machine Learning
Sep 30
Beyond Selection: Token Parameterization for Extreme Visual Token Compression
arXiv:2609.35232v2 Announce Type: replace-cross Abstract: Visual-token compression is effective for improving the efficiency of vision-language models, but under extreme compression budgets, token pr...
arXiv AI
Sep 2
SinkPruner: Sink-Free Visual Token Pruning for Multimodal Large Language Models
SinkPruner is a training‑free framework that prunes visual tokens for multimodal large language models by first removing high‑norm redundant tokens with a visual sanitizer and then selectively keeping tokens that align with the text query using a text‑guided pruner. The coarse‑to‑fine design reduces attention sink and dispersion, enabling an 89% token reduction while preserving 96.5% of LLaVA‑1.5’s performance and 91.8% of Qwen2.5‑VL’s performance across twelve image‑language and four video‑language benchmarks. The visual sanitizer also improves existing pruning methods, showing strong transferability.