arXiv:2608. 06628v1 Announce Type: new Abstract: Diffusion language models (dLLMs) offer a promising alternative to autoregressive models by accelerating inference through parallel decoding.
By Jinha Kim, Younghun Roh, Jaeyeon Kim
arXiv:2606.10537v2 Announce Type: replace
Abstract: Diffusion large language models (dLLMs) re-encode the entire prefix at every denoising step, causing recomputation that scales
quadratically with...
By Jing Xiong, Qi Han, Shansan Gong, Yunta Hsieh, Boyuan Zheng, Chengyue Wu, Chaofan Tao, Chenyang Zhao, Ngai Wong
DRAMA is a training‑free acceleration framework for multimodal diffusion large language models that dissects attention redundancy by probing attention tensors along spatial and temporal axes. It identifies two key redundancy patterns: a sparse subset of visual tokens consistently receives most attention, and attention distributions stabilize quickly after an initial phase. Leveraging these insights, DRAMA introduces Spatial‑Redundancy‑Aware Visual Token Pruning and Sufficiency‑Guided Adaptive Stopping, achieving an average 18.67× inference speedup across six vision‑language benchmarks while preserving competitive accuracy.
By Yi-Hsin Hung, Fangfu Liu, Shao-Yuan Lo, Chu-Song Chen
Diffusion large language models (dLLMs) re-encode the entire prefix at every denoising step, causing recomputation that scales quadratically with context length and becomes prohibitive for long-context scenarios. We propose Prefilling-dLLM, a training-free prefill-decode disaggregation framework for dLLMs that partitions the prefix into N chunks, caches their KV representations once, and selects the top-K most relevant chunks with intra-chunk token sparsity for decoding, showing that sparse prefilling can outperform dense attention while reducing per-step complexity from quadratic in the full sequence length to quadratic only in the decode length.
arXiv:2610.31911v1 Announce Type: new
Abstract: As large language models (LLMs) process increasingly long sequences, attention becomes a major computational bottleneck. Sparse attention mostly target...
By Keqi Deng, Ruchao Fan, Jinyu Li
Prism Attention introduces a proposal‑refined sparse attention framework that replaces static cross‑layer index reuse with a coarse‑to‑fine strategy, allowing layer‑wise adaptability while maintaining efficiency. The method leverages the observation that high‑attention tokens remain largely stable across layers, with changes concentrated near the boundary of previously selected regions. Experiments demonstrate state‑of‑the‑art performance on reasoning and long‑context benchmarks, achieving a 3.4× decoding speedup over FlashAttention at 128K context length.
By Zhenxu Tian, Kebin Liu, Zhengwu Yang, Yi Su, Qingqing Dang, Yanlin Sha, Kaipeng Deng, Dianhai Yu, Yanjun Ma, Juntao Li, Min Zhang