← Back to all news
arXiv Machine Learning

Gated Slot Attention-2: Two-Sided Associative Memory Correction in Linear Attention

The paper introduces Gated Slot Attention-2 (GSA2), a new linear‑attention architecture that combines a Gated Oja Rule‑2 for key‑side memory correction with a Gated Delta Rule‑2 for value‑side correction, both operating through shared latent slots. It also presents a hardware‑efficient chunkwise algorithm for parallel training. Experiments show that GSA2 consistently outperforms strong linear‑attention baselines while maintaining linear‑time sequence modeling and constant‑memory recurrent decoding.

Related stories

arXiv Machine Learning
Oct 7

RAM-Net: Linear-Time Sequence Modeling with Sparsely Addressable State

RAM‑Net introduces a linear‑time sequence model that replaces dense access to a shared recurrent state with sparse, address‑based access. The model arranges the state as a fixed‑size array of independent slots and uses an Address Decoder to map each token’s key or query to a small subset of slots, thereby reducing inter‑token interference. Empirical results show that RAM‑Net outperforms strong recurrent baselines on fine‑grained long‑range retrieval, achieves the lowest perplexity in commonsense reasoning, and accesses far fewer state elements per step than competing models.

By Kaicheng Xiao, Haotian Li, Liran Dong, Guoliang Xing
More like this →
arXiv Machine Learning
Sep 24

Attention Routing Stabilizes Early: Working-Set Inference for Recurrent Language Models

The paper investigates how attention dynamics evolve across recurrent depth in language models, finding that attention support stabilizes early while hidden states and outputs take longer. It proposes WISE, a training‑free method that uses full attention in early steps and then reuses the discovered sparse working set for later steps, preserving performance on multi‑hop QA tasks. Experiments show that WISE maintains quality up to 2K context, offers measurable speedups, and highlights the importance of recurrent discovery of attention support.

By Ke Wan, Chen Chen
llmsbenchmarks
More like this →
arXiv AI
Jun 10

Dynamic Linear Attention

arXiv:2606. 10650v1 Announce Type: cross Abstract: The scalability of Large Language Models (LLMs) to long contexts is fundamentally constrained by the quadratic complexity of standard attention, motivating the adoption of linear attention mechanisms with sub-quadratic cost.

By Xin Wang, Hui Shen, Boyuan Zheng, Xueshen Liu, Minkyoung Cho, Zhongwei Wan, Zesen Zhao, Zhuoqing Mao, Shen Yan, Mi Zhang
llmsbenchmarks
More like this →
Hugging Face Trending Papers
Jun 9

Dynamic Linear Attention

The scalability of Large Language Models (LLMs) to long contexts is fundamentally constrained by the quadratic complexity of standard attention, motivating the adoption of linear attention mechanisms with sub-quadratic cost. To improve representation capacity under long contexts, recent approaches organize memory in a multi-state manner.

llmsbenchmarks
More like this →
arXiv Machine Learning
Oct 21

Interdomain Attention: Beyond Token-Level Key-Value Memory

The paper introduces Interdomain Attention (IA), a hybrid architecture that blends softmax attention with recurrent memory within a single layer, leveraging a kernel regression perspective to compress keys and values into a state‑space model (SSM) memory. IA reconstructs the attention matrix implicitly from memory states, preserving the computational efficiency of SSMs while achieving performance gains. Experiments on models ranging from 125 M to 1.3 B parameters show IA surpasses an S4D baseline at matched memory size, outperforms standard Transformers on perplexity and common‑sense reasoning, and matches or exceeds SSM‑based models on genomic downstream tasks.

By Naoki Kiyohara, Harrison Bo Hua Zhu, Riccardo El Hassanin, Zhuo Sun, Wenlong Chen, Samir Bhatt, Yingzhen Li
llmsbenchmarks
More like this →