Joint Subspace Sparse Attention (JSSA) is a new method that reduces KV‑cache traffic in long‑context LLM decoding by selecting tokens at the token level while preserving query–key structure. It learns a shared low‑dimensional query–key subspace through a tractable spectral solution and implements a gather‑free indexed‑attention kernel that supports per‑KV‑head selections. Across various attention architectures, JSSA maintains near full‑KV accuracy, avoids generation‑length inflation, and can achieve up to 6.43× decoding throughput on a single GPU for 128K context, even at 700B‑parameter scale.
By Jiwon Song, Yoongon Kim, Jae-Joon Kim
arXiv:2605. 01910v2 Announce Type: replace-cross Abstract: Autoregressive decoding becomes bandwidth-limited at long contexts, as generating each token requires reading all $n_k$ key and value vectors from KV cache.
By Kyle Lee, Corentin Delacour, Kevin Callahan-Coray, Kyle Jiang, Can Yaras, Samet Oymak, Tathagata Srimani, Kerem Y. Camsari
arXiv:2606. 04511v1 Announce Type: cross Abstract: Sparse attention reduces compute and memory bandwidth for long-context LLM inference.
By Yaosheng Fu, Guangxuan Xiao, Xin Dong, Song Han, Oreste Villa
arXiv:2606. 13392v1 Announce Type: new Abstract: Ultra-long-context capability is becoming indispensable for frontier LLMs: agentic workflows, repository-scale code reasoning, and persistent memory all require the model to jointly attend over hundreds of thousands to millions of tokens, yet the quadratic cost of softmax attention makes this untenable at deployment scale.
By Xunhao Lai, Weiqi Xu, Yufeng Yang, Qiaorui Chen, Yang Xu, Lunbin Zeng, Xiaolong Li, Haohai Sun, Haichao Zhu, Vito Zhang, Pengyu Zhao
arXiv:2606. 06467v1 Announce Type: cross Abstract: Long-context inference in modern LLMs is increasingly constrained by decoding efficiency, especially in reasoning-heavy settings where models generate long intermediate chains of thought.
By Yutao Sun, Yanqi Zhang, Li Dong, Jianyong Wang, Furu Wei
arXiv:2605. 16928v2 Announce Type: replace-cross Abstract: Long-context inference in large language models is bottlenecked by the quadratic cost of full attention.
By Yanke Zhou, Yiduo Li, Hanlin Tang, Maohua Li, Kan Liu, Tao Lan, Lin Qu, Yuan Yao, Xiaoxing Ma
arXiv:2508. 18224v3 Announce Type: replace-cross Abstract: Recent advances in sparse attention mechanisms have demonstrated strong potential for reducing the computational cost of long-context training and inference in large language models (LLMs).
By Ran Yan, Youhe Jiang, Zhuoming Chen, Haohui Mai, Beidi Chen, Binhang Yuan
arXiv:2606. 09079v1 Announce Type: cross Abstract: Conventional LLMs keep the full KV cache loaded during decoding, causing a severe GPU memory bottleneck for ultra-long context serving.
By Yan Wang, Qifan Zhang, Jiachen Yu, Tian Liang, Dongyang Ma, Xiang Hu, Zibo Lin, Chunyang Li, Zhichao Wang, Jia Li, Yujiu Yang, Haitao Mi, Dong Yu
Elastic Threshold Attention (ETA) is a trainable attention mechanism that dynamically predicts contextual thresholds from query representations, enabling selective pruning of KV cache tokens during long‑context decoding. By multiplicatively suppressing sub‑threshold logits during training, ETA avoids representation collapse and eliminates localized attention sinks, allowing a 1.45B model to match dense attention performance at roughly 85% training sparsity and 38% active decode density. At inference, a custom Triton kernel achieves up to 2.5× faster decoding on sequences up to 512K tokens, and an offline calibration step can further reduce compute by 27% by freezing per‑head thresholds.
By Themistoklis Haris, Henry Li, Maryam Karimzadehgan
The paper introduces Stratified Sparse Attention (SSA), a framework that blends deterministic top‑k retrieval with score‑guided sampling to reduce the gap between sparse and dense attention in large language models. SSA attends to high‑scoring tokens deterministically while sampling the remaining attention mass using proxy scores, thereby recovering more of the attention distribution. Experiments on benchmarks such as RULER, LongBench, LOFT, and AIME show that SSA outperforms existing sparse methods across models ranging from 3B to 27B parameters, achieving significant accuracy gains and up to 2.96× decode speedup with optimized CUDA kernels.
By Sahil Joshi, Prithvi Dixit, Aryan Katiyar, Agniva Chowdhury, Joseph E. Gonzalez, Anshumali Shrivastava, Aditya Desai, Kumar Krishna Agrawal
CompKV introduces a compensation‑aware sparse attention framework for long‑context LLM inference. It partitions tokens into blocks and optimizes token selection to minimize the error introduced by block‑level mean compensation, using compact block‑level statistics. Experiments on RULER and LongBench‑Pro show CompKV outperforms other sparse baselines and achieves up to a 6.85× speedup over full attention.
By Zhen Huang, Ruizhe Yao, Danyi Liu, Xinrui Chen, Shuwei Li, Siru Zhong, Zijian Cao, Yushan Lai, Mingming Guo, Weijie Zheng, Haohuan Fu
The paper introduces Pre-hoc Sparsity (PrHS), a method that selects key-value (KV) cache entries before attention scoring to avoid posterior bias in large language model inference. By bounding mutual‑information loss through the dropped attention mass, PrHS offers explicit accuracy control and implements three orthogonal selectors across time, depth, and layer. Experiments on LLaMA and Mistral models show that PrHS cuts retrieval overhead by over 90%, achieves higher sparsity than HShare, and delivers significant speedups and reduced FLOPs on NVIDIA A100 GPUs while maintaining near‑dense accuracy.
By Yifei Gao, Lei Wang, Rong-Cheng Tu, Qixin Zhang, Jun Cheng, Dacheng Tao