DeltaLLM is a training‑free framework that accelerates on‑device large language model inference by introducing delta‑key sparse attention and a vectorized structured sparse online softmax. It reuses delta keys across query heads, preserves exact attention near the diagonal while approximating non‑local regions, and exploits hardware‑friendly sparsity with a specialized vector‑processor architecture. Experiments on BitNet‑b1.58‑2B‑4T and LLaMA3.2‑1B‑Instruct show up to 60% delta‑key sparsity, 63% structured attention‑score sparsity, and a 1.62× speedup on the KV260 FPGA with minimal accuracy loss.
By Jiawen Qi, Jiayu Liu, Chang Gao, Zhaochun Ren, Qinyu Chen
arXiv:2608.30439v1 Announce Type: cross
Abstract: Inference with transformer-based large language models (LLMs) is often limited by the memory-bound KV cache and quadratic attention cost. State-space...
By Simon Richter, Ruhai Lin, Jason Yik, Taylor Kergan, Rui-Jie Zhu, Farshad Moradi, Jason Eshraghian
arXiv:2608. 15533v1 Announce Type: cross Abstract: Linear attention models eliminate the quadratic prefix computation and context-growing KV cache of softmax attention by replacing pairwise token interactions with recurrent state updates.
By Junqing Lin, Jingwei Sun, Guangzhong Sun
arXiv:2609.38166v1 Announce Type: cross
Abstract: Recent LLMs increasingly adopt hybrid designs that replace standard attention with linear attention, such as Gated DeltaNet (GDN) and Kimi Delta Atte...
By Yi Pan, Haocheng Xi, Kan Zhu, Xingyang Li, Yibo Wu, Mayank Mishra, Hongtao Zhang, William X. Zheng, Baris Kasikci, Song Han, Kurt Keutzer, Rishabh Iyer, Ion Stoica
arXiv:2606. 01117v1 Announce Type: cross Abstract: Extreme multi-label classification (XMC) involves learning models over large output spaces with millions of labels, making the output layer a memory-compute bottleneck.
By Nasib Ullah, Jinbin Zhang, Jean Lucien Randrianantenaina, Erik Schultheis, Rohit Babbar
arXiv:2608. 19758v1 Announce Type: new Abstract: Long-context modeling is a pivotal capability for Large Language Models, yet the quadratic complexity of attention remains a critical bottleneck, particularly during the compute-intensive prefilling phase.
By Qihang Fan, Huaibo Huang, Zhiying Wu, Bingning Wang, Ran He
arXiv:2609.38169v1 Announce Type: cross
Abstract: Linear attention replaces growing KV caches with fixed-size recurrent states, yet these persistent states can become a substantial memory bottleneck...
By Bingchen Yao, Haobo Xu, Haokun Lin, Yichen Wu, Ziyu Guo, Renrui Zhang, Zhichao Lu, Zhenan Sun, Ying Wei
arXiv:2606. 27229v1 Announce Type: cross Abstract: Recurrent models must forget in order to remember, yet the state of the art decides what to erase without consulting what is stored -- the gate sees only the arriving token, not the memory it is about to modify.
By Sayak Dutta
arXiv:2610.20503v1 Announce Type: new
Abstract: Speculative decoding accelerates autoregressive generation by drafting multiple tokens per step, yet feature-based drafters rely on target hidden featu...
By Bohan Sun, Zunhai Su, Xialie Zhuang, Jianjin Li, Ngai Wong, Chuan-Wei Kuo
arXiv:2610.15715v1 Announce Type: new
Abstract: Linear attention promises the efficiency needed for long-context language modelling, yet its fixed-size recurrent state must simultaneously support ong...
By Li Nanbo, Firas Laakom, Yilan Zhang, Piotr Pi\k{e}kos, Yimeng Chen, Liangyu Wang, Yifan Xu, Mateusz Ostaszewski, Dandan Guo, Wenyi Wang, Yanning Dai, Yuhui Wang, J\"{u}rgen Schmidhuber
STAPLE is a method that integrates static sparse memory into pretrained language models by adding token‑indexed embeddings as a residual pathway while preserving the original feed‑forward network. On the OLMo‑2 1B and 7B models, STAPLE outperforms dense training and other static sparse baselines, improving benchmark scores by up to 4.27 points and achieving faster decoding with less GPU memory and energy usage. The approach demonstrates that large pretrained models can be expanded efficiently without costly sparse pretraining from scratch.
By Ranajoy Sadhukhan, Euijin Hong, Sheng Cao, Zechun Liu, Shuowei Jin, Sha Li, Changlong Yu, Xin Liu, Qin Lu, Zhenyu Shi, Priyanka Nigam, Beidi Chen
The paper introduces the Global State Model (GSM), a causal encoder-decoder architecture that separates historical information acquisition from deep computation. GSM’s encoder aggregates long-range information via multi-stage sparse retrieval into a fixed-size shared state, while the decoder transforms this state without repeated historical access, making attention and cache costs independent of history length. Experiments show GSM matches or exceeds the performance of DeepSeek-V4.1-Flash baselines across scales, while improving training and inference throughput and reducing cache overhead.
By Yunao Zheng, Bin Wen, Xiaojie Wang, Kaiyu Jiang, Xuanyu Zheng, Changyi Liu, Hongyi Fu, Jianxiong Wang, Tianke Zhang, Haonan Fan, Yingxin Li, Jiankang Chen, Xu Wang, Tingting Gao, Han Li