arXiv Computation and Language By Zhichen Zeng, Xichong Zhang, Junpan Wu, Yifei Zuo, Chi-Chih Chang, Jiayi Wang, Maohua Nie, Ji Liu, Ang Li, Banghua Zhu

Step-dLLM: Adaptive Step-aware Sparse Attention for Efficient Diffusion LLM Inference

Read the original on arXiv Computation and Language →

Step-dLLM introduces an adaptive, step-aware sparse attention mechanism for diffusion large language models (dLLMs). It refreshes a block-sparse mask at specific anchor steps along the denoising trajectory and applies a head-adaptive global sparsity budget, allowing efficient reuse of attention patterns within locally consistent intervals. The method, implemented with a fused IO-aware Triton kernel, achieves a 2.78× speedup over dense FlashAttention on LLaDA-1.5 while improving the accuracy‑latency trade‑off compared to prior sparse‑attention approaches.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computation and Language.

arXiv Computation and Language
Sep 1

Prefilling-dLLM: Predictive Prefilling for Long-Context Inference in Diffusion Language Models

arXiv:2606.10537v2 Announce Type: replace Abstract: Diffusion large language models (dLLMs) re-encode the entire prefix at every denoising step, causing recomputation that scales quadratically with...

By Jing Xiong, Qi Han, Shansan Gong, Yunta Hsieh, Boyuan Zheng, Chengyue Wu, Chaofan Tao, Chenyang Zhao, Ngai Wong
arXiv AI
3d ago

DRAMA: Dissecting Attention Redundancy for Accelerating Multimodal Diffusion Large Language Models

DRAMA is a training‑free acceleration framework for multimodal diffusion large language models that dissects attention redundancy by probing attention tensors along spatial and temporal axes. It identifies two key redundancy patterns: a sparse subset of visual tokens consistently receives most attention, and attention distributions stabilize quickly after an initial phase. Leveraging these insights, DRAMA introduces Spatial‑Redundancy‑Aware Visual Token Pruning and Sufficiency‑Guided Adaptive Stopping, achieving an average 18.67× inference speedup across six vision‑language benchmarks while preserving competitive accuracy.

By Yi-Hsin Hung, Fangfu Liu, Shao-Yuan Lo, Chu-Song Chen
Hugging Face Trending Papers
Jun 9

Prefilling-dLLM: Predictive Prefilling for Long-Context Inference in Diffusion Language Models

Diffusion large language models (dLLMs) re-encode the entire prefix at every denoising step, causing recomputation that scales quadratically with context length and becomes prohibitive for long-context scenarios. We propose Prefilling-dLLM, a training-free prefill-decode disaggregation framework for dLLMs that partitions the prefix into N chunks, caches their KV representations once, and selects the top-K most relevant chunks with intra-chunk token sparsity for decoding, showing that sparse prefilling can outperform dense attention while reducing per-step complexity from quadratic in the full sequence length to quadratic only in the decode length.

arXiv AI
Oct 19

Prism Attention: Proposal-Refined Index Sharing Mechanism for Efficient LLMs Inference

Prism Attention introduces a proposal‑refined sparse attention framework that replaces static cross‑layer index reuse with a coarse‑to‑fine strategy, allowing layer‑wise adaptability while maintaining efficiency. The method leverages the observation that high‑attention tokens remain largely stable across layers, with changes concentrated near the boundary of previously selected regions. Experiments demonstrate state‑of‑the‑art performance on reasoning and long‑context benchmarks, achieving a 3.4× decoding speedup over FlashAttention at 128K context length.

By Zhenxu Tian, Kebin Liu, Zhengwu Yang, Yi Su, Qingqing Dang, Yanlin Sha, Kaipeng Deng, Dianhai Yu, Yanjun Ma, Juntao Li, Min Zhang