Multi-State Linear Attention (MSLA) improves long‑sequence modeling by compressing context into separate memory states, but its aggregation only uses mean alignment, ignoring distribution within segments. MERLIN (MomEnt Routing for multi‑state LINear attention) enhances this by tracking first and second moments of each segment’s keys, enabling queries to recover both mean and variance of alignment for more content‑aware aggregation. Experiments on Gated DeltaNet‑1.3B and Mamba‑2‑780M demonstrate that MERLIN consistently outperforms existing aggregation methods on long‑context tasks, achieving significant accuracy gains while using less memory.
By Hee Seung Hwang, Sanghyuk Chun, Xindi Wu, Olga Russakovsky, Zhiwei Deng
arXiv:2609.39661v1 Announce Type: new
Abstract: Self-attention gives LLMs fine-grained, query-dependent access to context, but dense token interactions incur quadratic prefill cost and a key--value c...
By Zhentao Tan, Jingyi Shen, Yanbo Li, Yao Liu, Yue Wu, Jieping Ye
MS-GLA introduces Multi-Scale Gated Linear Attention, distributing attention heads across multiple temporal resolutions to address representational bottlenecks in GLA Transformers. Coarser resolutions pool longer token spans for long‑range dependencies, while finer heads capture local syntax, and a learnable fusion layer recombines outputs dynamically. Experiments on language modeling, recall‑intensive tasks, and long‑context generalization show MS‑GLA outperforms GLA with up to 18.9% improvement on recall tasks and 9.5% lower perplexity on language modeling benchmarks.
By Prasoon Dev, Anirudh Sankar, Vasudeva Varma
arXiv:2606. 01294v1 Announce Type: cross Abstract: Linear attention reduces the quadratic cost of softmax attention by maintaining a recurrent fast-weight state, but it consistently lags on in-context retrieval and long-context tasks.
By Dong Le, Thong Nguyen, Cong-Duy Nguyen, Anh Tuan Luu
arXiv:2608.30310v1 Announce Type: cross
Abstract: Hybrid large language models interleave full-attention layers with linear-attention layers to reduce the cost of long-context inference. This structu...
By Yirui Liu, Ruoling Qi, Xuaner Wu, Penghang Liu, Jian Chen
The scalability of Large Language Models (LLMs) to long contexts is fundamentally constrained by the quadratic complexity of standard attention, motivating the adoption of linear attention mechanisms with sub-quadratic cost. To improve representation capacity under long contexts, recent approaches organize memory in a multi-state manner.
arXiv:2608.28444v1 Announce Type: cross
Abstract: Due to the nature of quadratic attention, Large Language Models (LLMs) consume a lot of memory and energy. Every new token costs more than the previo...
By Alexia Jolicoeur-Martineau, Rhea Sanjay Sukthanker, Pashmina Cameron, Emy Gervais
arXiv:2606. 16093v1 Announce Type: cross Abstract: Modeling long-range dependencies remains a central challenge in natural language processing.
By Kuzey Torlak, H\"useyin Arda Arslan, An{\i}l Dervi\c{s}o\u{g}lu, Beyza Nur Deniz, Onur Boyar
arXiv:2607. 02980v1 Announce Type: cross Abstract: Scaling modern large language models (LLMs) to long contexts is limited by the quadratic computation cost, and poor length extrapolation of dense attention.
By Xiang Hu, Xinyu Wei, Hao Gu, Minshen Zhang, Tian Liang, Huayang Li, Lei Zhu, Yan Wang, Sirui Han, Yushi Bai, Kewei Tu, Haitao Mi, Leo Liang
arXiv:2606. 10650v1 Announce Type: cross Abstract: The scalability of Large Language Models (LLMs) to long contexts is fundamentally constrained by the quadratic complexity of standard attention, motivating the adoption of linear attention mechanisms with sub-quadratic cost.
By Xin Wang, Hui Shen, Boyuan Zheng, Xueshen Liu, Minkyoung Cho, Zhongwei Wan, Zesen Zhao, Zhuoqing Mao, Shen Yan, Mi Zhang
arXiv:2610.14754v1 Announce Type: new
Abstract: Hybrid language models interleave Gated DeltaNet (GDN) layers, whose recurrent memory has a fixed size, with full-attention layers, whose key-value (KV...
By Junjie Li, Jiong Lou, Jie Li
arXiv:2602.04852v3 Announce Type: replace
Abstract: Linear attention offers a computationally efficient yet expressive alternative to softmax attention. However, recent empirical results indicate tha...
By Philipp Nazari, T. Konstantin Rusch