arXiv Computation and Language By Siyuan Yan, Shen Gu, Xiaoxing Ma, Chun Cao, Jingwei Xu

DUET: Dual-Sparse Inference for Block Diffusion Language Models

Read the original on arXiv Computation and Language →

DUET is a training‑free dual‑sparse inference framework for diffusion language models that reduces inference cost by two techniques: prefix Key‑Value sparsity and early query sparsity. It constructs a compact historical KV subset reused across denoising steps, and predicts which positions will be updated to propagate only the necessary queries through the suffix Transformer layers. Experiments demonstrate that DUET keeps accuracy while delivering up to 4.72× speedups on large models with 32K context length.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computation and Language.

arXiv Computation and Language
Nov 4

Step-dLLM: Adaptive Step-aware Sparse Attention for Efficient Diffusion LLM Inference

Step-dLLM introduces an adaptive, step-aware sparse attention mechanism for diffusion large language models (dLLMs). It refreshes a block-sparse mask at specific anchor steps along the denoising trajectory and applies a head-adaptive global sparsity budget, allowing efficient reuse of attention patterns within locally consistent intervals. The method, implemented with a fused IO-aware Triton kernel, achieves a 2.78× speedup over dense FlashAttention on LLaDA-1.5 while improving the accuracy‑latency trade‑off compared to prior sparse‑attention approaches.

By Zhichen Zeng, Xichong Zhang, Junpan Wu, Yifei Zuo, Chi-Chih Chang, Jiayi Wang, Maohua Nie, Ji Liu, Ang Li, Banghua Zhu
arXiv Computation and Language
Aug 25

Accelerating Diffusion Language Models via Structured Suffix Modeling

The paper introduces a training‑free structured suffix modeling technique to accelerate Diffusion Language Models (DLMs). It partitions the suffix into local, middle, and tail regions, retaining varying numbers of tokens per region and incorporating previous decoding results into current token representations. Experiments on three DLMs show significant speedups—up to 72.81× in long‑sequence inference—while often improving performance, and the method is compatible with existing acceleration strategies.

By Zifeng Cheng, Keda Li, Zhiwei Jiang, Cong Wang, Fei Shen, Qing Gu
Hugging Face Trending Papers
Oct 13

Prefilling-dLLM v2: Learned Chunk Routing for Long-Context Diffusion Language Models

Prefilling-dLLM v2 is a diffusion language model that learns to route computation using a native mask token, enabling efficient sparse attention across long contexts. By appending a mask at the end of each chunk and using a learned role offset, the model selectively attends to top‑K chunks every fourth layer while maintaining local windows elsewhere. Evaluated on 16K–128K contexts, Prefilling-dLLM v2 outperforms other sparse methods (NSA, DSA, SWA) on RULER tasks and achieves a 172× speedup over full dense attention at 128K tokens.