← Back to all news
arXiv Machine Learning By Ruijie Li, Shengnan Ding, Weimin Zhang, Derick Tang, Zhanpeng Zeng, Qinsong Zeng, Ming Chen, Jiaxi Hu, Yuxuan Liang

Gated Slot Attention-2: Two-Sided Associative Memory Correction in Linear Attention

Read the original on arXiv Machine Learning →

The paper introduces Gated Slot Attention-2 (GSA2), a new linear‑attention architecture that combines a Gated Oja Rule‑2 for key‑side memory correction with a Gated Delta Rule‑2 for value‑side correction, both operating through shared latent slots. It also presents a hardware‑efficient chunkwise algorithm for parallel training. Experiments show that GSA2 consistently outperforms strong linear‑attention baselines while maintaining linear‑time sequence modeling and constant‑memory recurrent decoding.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

Related stories

arXiv Machine Learning
Oct 7

RAM-Net: Linear-Time Sequence Modeling with Sparsely Addressable State

RAM‑Net introduces a linear‑time sequence model that replaces dense access to a shared recurrent state with sparse, address‑based access. The model arranges the state as a fixed‑size array of independent slots and uses an Address Decoder to map each token’s key or query to a small subset of slots, thereby reducing inter‑token interference. Empirical results show that RAM‑Net outperforms strong recurrent baselines on fine‑grained long‑range retrieval, achieves the lowest perplexity in commonsense reasoning, and accesses far fewer state elements per step than competing models.

By Kaicheng Xiao, Haotian Li, Liran Dong, Guoliang Xing
More like this →
arXiv Machine Learning
Sep 24

Attention Routing Stabilizes Early: Working-Set Inference for Recurrent Language Models

The paper investigates how attention dynamics evolve across recurrent depth in language models, finding that attention support stabilizes early while hidden states and outputs take longer. It proposes WISE, a training‑free method that uses full attention in early steps and then reuses the discovered sparse working set for later steps, preserving performance on multi‑hop QA tasks. Experiments show that WISE maintains quality up to 2K context, offers measurable speedups, and highlights the importance of recurrent discovery of attention support.

By Ke Wan, Chen Chen
llmsbenchmarks
More like this →
arXiv AI
Jun 10

Dynamic Linear Attention

arXiv:2606. 10650v1 Announce Type: cross Abstract: The scalability of Large Language Models (LLMs) to long contexts is fundamentally constrained by the quadratic complexity of standard attention, motivating the adoption of linear attention mechanisms with sub-quadratic cost.

By Xin Wang, Hui Shen, Boyuan Zheng, Xueshen Liu, Minkyoung Cho, Zhongwei Wan, Zesen Zhao, Zhuoqing Mao, Shen Yan, Mi Zhang
llmsbenchmarks
More like this →