arXiv:2606. 01294v1 Announce Type: cross Abstract: Linear attention reduces the quadratic cost of softmax attention by maintaining a recurrent fast-weight state, but it consistently lags on in-context retrieval and long-context tasks.
By Dong Le, Thong Nguyen, Cong-Duy Nguyen, Anh Tuan Luu
arXiv:2607. 20457v1 Announce Type: cross Abstract: Inference with large language models (LLMs) on long sequences is computationally expensive due to the quadratic complexity of self-attention.
By Aryan Sood, Shantanu Acharya
arXiv:2602. 03681v2 Announce Type: replace-cross Abstract: The quadratic computational complexity of softmax transformers has become a bottleneck in long-context scenarios.
By Difan Deng, Andreas Bentzen Winje, Lukas Fehring, Marius Lindauer
arXiv:2602.04852v3 Announce Type: replace
Abstract: Linear attention offers a computationally efficient yet expressive alternative to softmax attention. However, recent empirical results indicate tha...
By Philipp Nazari, T. Konstantin Rusch
arXiv:2606. 02332v1 Announce Type: new Abstract: Combining attention's global retrieval with the sequential importance signal of state space models (SSMs) is the open challenge of hybrid language modeling.
By Soohyeong Shin, Yeongwook Yang
arXiv:2609.13141v1 Announce Type: new
Abstract: Post-training attention sparsification reduces the quadratic cumulative attention cost of pretrained Transformers by selecting a small set of context u...
By Zhiwei Li, Lei Zhu, Hao Gu, Xiang Hu, Yan Wang, Haitao Mi, Sirui Han, Leo Liang, Zhijiang Guo