arXiv AI

DRAMA: Dissecting Attention Redundancy for Accelerating Multimodal Diffusion Large Language Models

DRAMA is a training‑free acceleration framework for multimodal diffusion large language models that dissects attention redundancy by probing attention tensors along spatial and temporal axes. It identifies two key redundancy patterns: a sparse subset of visual tokens consistently receives most attention, and attention distributions stabilize quickly after an initial phase. Leveraging these insights, DRAMA introduces Spatial‑Redundancy‑Aware Visual Token Pruning and Sufficiency‑Guided Adaptive Stopping, achieving an average 18.67× inference speedup across six vision‑language benchmarks while preserving competitive accuracy.

arXiv Computation and Language
Nov 4

Step-dLLM: Adaptive Step-aware Sparse Attention for Efficient Diffusion LLM Inference

Step-dLLM introduces an adaptive, step-aware sparse attention mechanism for diffusion large language models (dLLMs). It refreshes a block-sparse mask at specific anchor steps along the denoising trajectory and applies a head-adaptive global sparsity budget, allowing efficient reuse of attention patterns within locally consistent intervals. The method, implemented with a fused IO-aware Triton kernel, achieves a 2.78× speedup over dense FlashAttention on LLaDA-1.5 while improving the accuracy‑latency trade‑off compared to prior sparse‑attention approaches.

By Zhichen Zeng, Xichong Zhang, Junpan Wu, Yifei Zuo, Chi-Chih Chang, Jiayi Wang, Maohua Nie, Ji Liu, Ang Li, Banghua Zhu
arXiv Computation and Language
Aug 28

Visual Information-Guided Parallel Decoding for Diffusion Multimodal Large Language Models

Visual Information-Guided Parallel Decoding for Diffusion Multimodal Large Language Models introduces the VIG‑Sampler, a method that prioritizes tokens for decoding based on their attention to image tokens and penalizes redundancy in image‑attention distributions. The approach aims to improve the quality of multimodal generation by selecting more informative tokens during diffusion decoding. Experiments on seven captioning and VQA benchmarks with three open‑source dMLLMs show that VIG‑Sampler outperforms the Info‑Gain Sampler by an average of 19.3 CIDEr points and achieves better COCO Caption results using only half as many decoding steps.

By Insu Lee, Wooje Park, Wonseok Shin, Jinwoo Son, Byonghyo Shim
arXiv AI
Oct 31

PACE: Pareto-Adaptive Compression for Efficient Native MLLMs

The paper introduces PACE, a Pareto‑adaptive compression framework designed for native multimodal large language models (MLLMs) that directly tokenize pixel patches without external vision encoders. By analyzing token discriminability and heterogeneous redundancy across spatial dimensions, PACE decomposes attention into height‑ and width‑specific saliency maps and applies a Pareto‑optimization strategy to allocate token budgets, achieving near‑lossless performance with only 25‑50% of visual tokens and more than 2× FLOP savings. Experiments demonstrate that PACE not only preserves but sometimes improves benchmark results compared to existing ViT‑based compression methods.

By Lei Tian, Chunzheng Zhu, Kai Wu, Yifan Zhang, Haihua Yang
arXiv Computer Vision
6d ago

RepFusion: Leveraging Multimodal Priors for Denoising in Representation Space

RepFusion leverages multimodal large language models (MLLMs) as noisy representation encoders to condition a diffusion transformer for denoising in representation space. By reusing MLLM outputs across denoising steps, the method achieves faster inference while maintaining performance comparable to per-step recomputation. Experiments show RepFusion outperforms baselines with similar inference budgets, demonstrating that MLLMs provide strong priors for denoising visual representations.

By Xichen Pan, Aashu Singh, Satya Narayan Shukla, Xiangjun Fan, Shlok Kumar Mishra, Saining Xie
arXiv AI
Sep 10

STAR-Pro: Stage-Wise Token Adaptive Reduction with Progressive Refinement for Efficient Large Vision-Language Models

arXiv:2609.05916v1 Announce Type: cross Abstract: Large vision-language models (LVLMs) achieve strong multimodal understanding, but the hundreds to thousands of visual tokens they process impose subs...

By Yichen Guo, Tinghao Wang, Qizhe Zhang, Lingbei Meng, Yuan Zhang, Jiajun Cao, Hao Jiang, Chenwei Wu, Jixian Wu, Sixiang Chen, Tao Luo, Hongyang Cheng, Kai Tang, Chenxi Li, Renyuan Li, Xiande Huang, Wenya Wang, Shanghang Zhang