DeltaLLM is a training‑free framework that accelerates on‑device large language model inference by introducing delta‑key sparse attention and a vectorized structured sparse online softmax. It reuses delta keys across query heads, preserves exact attention near the diagonal while approximating non‑local regions, and exploits hardware‑friendly sparsity with a specialized vector‑processor architecture. Experiments on BitNet‑b1.58‑2B‑4T and LLaMA3.2‑1B‑Instruct show up to 60% delta‑key sparsity, 63% structured attention‑score sparsity, and a 1.62× speedup on the KV260 FPGA with minimal accuracy loss.
By Jiawen Qi, Jiayu Liu, Chang Gao, Zhaochun Ren, Qinyu Chen
arXiv:2606. 04511v1 Announce Type: cross Abstract: Sparse attention reduces compute and memory bandwidth for long-context LLM inference.
By Yaosheng Fu, Guangxuan Xiao, Xin Dong, Song Han, Oreste Villa
AMMA is a multi‑chiplet, memory‑centric architecture designed to reduce latency in large‑context attention for language models. It replaces GPU compute dies with HBM‑PNM cubes, doubling memory bandwidth and employing a logic‑die microarchitecture, two‑level hybrid parallelism, and a reordered collective flow to minimize communication overhead. Design‑space exploration and evaluations show AMMA achieves 15.5× lower attention latency and 6.9× lower energy consumption compared with the NVIDIA H100.
By Zhongkai Yu, Haotian Ye, Chenyang Zhou, Ohm Rishabh Venkatachalam, Zaifeng Pan, Zhengding Hu, Junsung Kim, Won Woo Ro, Po-An Tsai, Shuyi Pei, Yangwook Kang, Yufei Ding
arXiv:2611.07256v1 Announce Type: cross
Abstract: Sparsity and quantization reduce the cost of attention by skipping interactions and lowering the precision of those retained. However, useful interac...
By Hanzhi Zhang, Qiao Zhang, Qinglei Cao, Heng Fan, Yan Huang, Kewei Sha, Yunhe Feng
The paper introduces QUILT, a new sparse‑attention execution mechanism that jointly processes neighboring queries to reuse shared key‑value (KV) entries, thereby reducing redundant memory traffic and computation. QUILT employs Shift‑and‑Compare Set Decomposition (SCSD) to convert irregular set operations into regular data‑parallel primitives and pipelines this with attention computation to hide overhead. The approach includes cascaded sharing across multiple granularities and a tile‑aware strategy that balances sharing with hardware utilization, achieving up to 55.1% lower kernel latency and 36.8% faster time‑to‑first‑token on LongBench benchmarks with negligible accuracy loss.
By Zhenduo Zhao, Qihui Zhou, Mingcong Song, Zhiyi Chen, Chuangguan Ye, Fengfan Hou, Zequn Gong, Jing Li, Hongjie Si, Guoping Long
The paper introduces INTRA, a hardware‑aware static sparse attention mechanism that interleaves non‑contiguous token groups to enable efficient long‑range communication while reducing quadratic cost. INTRA uses Scatter and Gather layers to create short mediated paths across layers, and a FlashAttention‑style kernel computes structured indices at runtime for dense packing of selected tokens. Experiments on an A100 GPU show significant speedups—up to 8.4× over MoBA and 10.4× over density‑constrained NSA—while maintaining or improving image quality and long‑context performance on models like FLUX.1‑dev and LLaMA‑3.1‑8B.
By Yuxuan Xia, Wenbo Lu, Shaoyi Zheng, Shengjie Wang