arXiv Machine Learning June 24, 2026 By Piotr Nawrot, Robert Li, Renjie Huang, Sebastian Ruder, Kelly Marchisio, Edoardo M. Ponti

The Sparse Frontier: Sparse Attention Trade-offs in Transformer LLMs

Read the original on arXiv Machine Learning →

arXiv:2504. 17768v3 Announce Type: replace-cross Abstract: Sparse attention offers a promising strategy to extend long-context capabilities in Transformer LLMs, yet its efficiency-accuracy trade-offs remain unclear due to the lack of comprehensive evaluation.

Summary generated by The Flow from the publisher's feed. The full article lives at arXiv Machine Learning.

arXiv AI

Jun 9

Full Attention Strikes Back: Transferring Full Attention into Sparse within Hundred Training Steps

arXiv:2605. 16928v2 Announce Type: replace-cross Abstract: Long-context inference in large language models is bottlenecked by the quadratic cost of full attention.

By Yanke Zhou, Yiduo Li, Hanlin Tang, Maohua Li, Kan Liu, Tao Lan, Lin Qu, Yuan Yao, Xiaoxing Ma

llms efficiency benchmarks

arXiv Machine Learning

Jul 2

FSA: An Alternative Efficient Implementation of Native Sparse Attention Kernel

arXiv:2508. 18224v3 Announce Type: replace-cross Abstract: Recent advances in sparse attention mechanisms have demonstrated strong potential for reducing the computational cost of long-context training and inference in large language models (LLMs).

By Ran Yan, Youhe Jiang, Zhuoming Chen, Haohui Mai, Beidi Chen, Binhang Yuan

llms benchmarks

arXiv AI

Jul 21

Lil: Less is Less When Applying Post-Training Sparse-Attention Algorithms in Long-Decode Stage

arXiv:2601. 03043v4 Announce Type: replace-cross Abstract: Large language models (LLMs) demonstrate strong capabilities across a wide range of complex tasks and are increasingly deployed at scale, placing significant demands on inference efficiency.

By Junhao Hu, Fangze Li, Mingtao Xu, Feifan Meng, Shiju Zhao, Tiancheng Hu, Ting Peng, Anmin Liu, Wenrui Huang, Chenxu Liu, Ziyue Hua, Tao Xie

llms benchmarks

arXiv AI

Jul 23

ELSAA: Efficient Low-Rank and Sparse Attention Approximation for Training Transformers

arXiv:2607. 20214v1 Announce Type: cross Abstract: The quadratic $N\times N$ attention score matrix remains a central obstacle to extending Transformers to longer input lengths.

By Mahdi Heidari, Mohammad Mahdi Rahimi, Jaekyun Moon

llms efficiency

arXiv Machine Learning

Jul 13

COBS: Cumulant Order Block Sparse Attention

arXiv:2607. 09052v1 Announce Type: new Abstract: Block sparse attention is a hardware friendly way to alleviate the key-value (KV) cache read bottleneck in large language models (LLMs).

By Alexander Tian, Aditya Ghai, Sanjit Neelam, Zaal Vasania, Akshay Mishra

llms efficiency benchmarks

arXiv AI

Jul 7

Hierarchical Sparse Attention Done Right: Toward Infinite Context Modeling

arXiv:2607. 02980v1 Announce Type: cross Abstract: Scaling modern large language models (LLMs) to long contexts is limited by the quadratic computation cost, and poor length extrapolation of dense attention.

By Xiang Hu, Xinyu Wei, Hao Gu, Minshen Zhang, Tian Liang, Huayang Li, Lei Zhu, Yan Wang, Sirui Han, Yushi Bai, Kewei Tu, Haitao Mi, Leo Liang

llms

Related stories

Full Attention Strikes Back: Transferring Full Attention into Sparse within Hundred Training Steps

FSA: An Alternative Efficient Implementation of Native Sparse Attention Kernel

Lil: Less is Less When Applying Post-Training Sparse-Attention Algorithms in Long-Decode Stage

ELSAA: Efficient Low-Rank and Sparse Attention Approximation for Training Transformers

COBS: Cumulant Order Block Sparse Attention

Hierarchical Sparse Attention Done Right: Toward Infinite Context Modeling