arXiv Computation and Language

Step-dLLM: Adaptive Step-aware Sparse Attention for Efficient Diffusion LLM Inference

Step-dLLM introduces an adaptive, step-aware sparse attention mechanism for diffusion large language models (dLLMs). It refreshes a block-sparse mask at specific anchor steps along the denoising trajectory and applies a head-adaptive global sparsity budget, allowing efficient reuse of attention patterns within locally consistent intervals. The method, implemented with a fused IO-aware Triton kernel, achieves a 2.78× speedup over dense FlashAttention on LLaDA-1.5 while improving the accuracy‑latency trade‑off compared to prior sparse‑attention approaches.

arXiv Computation and Language
Sep 1

Prefilling-dLLM: Predictive Prefilling for Long-Context Inference in Diffusion Language Models

arXiv:2606.10537v2 Announce Type: replace Abstract: Diffusion large language models (dLLMs) re-encode the entire prefix at every denoising step, causing recomputation that scales quadratically with...

By Jing Xiong, Qi Han, Shansan Gong, Yunta Hsieh, Boyuan Zheng, Chengyue Wu, Chaofan Tao, Chenyang Zhao, Ngai Wong
arXiv AI
3d ago

DRAMA: Dissecting Attention Redundancy for Accelerating Multimodal Diffusion Large Language Models

DRAMA is a training‑free acceleration framework for multimodal diffusion large language models that dissects attention redundancy by probing attention tensors along spatial and temporal axes. It identifies two key redundancy patterns: a sparse subset of visual tokens consistently receives most attention, and attention distributions stabilize quickly after an initial phase. Leveraging these insights, DRAMA introduces Spatial‑Redundancy‑Aware Visual Token Pruning and Sufficiency‑Guided Adaptive Stopping, achieving an average 18.67× inference speedup across six vision‑language benchmarks while preserving competitive accuracy.

By Yi-Hsin Hung, Fangfu Liu, Shao-Yuan Lo, Chu-Song Chen
Hugging Face Trending Papers
Jun 9

Prefilling-dLLM: Predictive Prefilling for Long-Context Inference in Diffusion Language Models

Diffusion large language models (dLLMs) re-encode the entire prefix at every denoising step, causing recomputation that scales quadratically with context length and becomes prohibitive for long-context scenarios. We propose Prefilling-dLLM, a training-free prefill-decode disaggregation framework for dLLMs that partitions the prefix into N chunks, caches their KV representations once, and selects the top-K most relevant chunks with intra-chunk token sparsity for decoding, showing that sparse prefilling can outperform dense attention while reducing per-step complexity from quadratic in the full sequence length to quadratic only in the decode length.

arXiv AI
Oct 19

Prism Attention: Proposal-Refined Index Sharing Mechanism for Efficient LLMs Inference

Prism Attention introduces a proposal‑refined sparse attention framework that replaces static cross‑layer index reuse with a coarse‑to‑fine strategy, allowing layer‑wise adaptability while maintaining efficiency. The method leverages the observation that high‑attention tokens remain largely stable across layers, with changes concentrated near the boundary of previously selected regions. Experiments demonstrate state‑of‑the‑art performance on reasoning and long‑context benchmarks, achieving a 3.4× decoding speedup over FlashAttention at 128K context length.

By Zhenxu Tian, Kebin Liu, Zhengwu Yang, Yi Su, Qingqing Dang, Yanlin Sha, Kaipeng Deng, Dianhai Yu, Yanjun Ma, Juntao Li, Min Zhang
Hugging Face Trending Papers
Oct 13

Prefilling-dLLM v2: Learned Chunk Routing for Long-Context Diffusion Language Models

Prefilling-dLLM v2 is a diffusion language model that learns to route computation using a native mask token, enabling efficient sparse attention across long contexts. By appending a mask at the end of each chunk and using a learned role offset, the model selectively attends to top‑K chunks every fourth layer while maintaining local windows elsewhere. Evaluated on 16K–128K contexts, Prefilling-dLLM v2 outperforms other sparse methods (NSA, DSA, SWA) on RULER tasks and achieves a 172× speedup over full dense attention at 128K tokens.

arXiv Computation and Language
Nov 9

DUET: Dual-Sparse Inference for Block Diffusion Language Models

DUET is a training‑free dual‑sparse inference framework for diffusion language models that reduces inference cost by two techniques: prefix Key‑Value sparsity and early query sparsity. It constructs a compact historical KV subset reused across denoising steps, and predicts which positions will be updated to propagate only the necessary queries through the suffix Transformer layers. Experiments demonstrate that DUET keeps accuracy while delivering up to 4.72× speedups on large models with 32K context length.

By Siyuan Yan, Shen Gu, Xiaoxing Ma, Chun Cao, Jingwei Xu