← Back to all news
arXiv Machine Learning By Ke Wan, Chen Chen

Attention Routing Stabilizes Early: Working-Set Inference for Recurrent Language Models

Read the original on arXiv Machine Learning →

The paper investigates how attention dynamics evolve across recurrent depth in language models, finding that attention support stabilizes early while hidden states and outputs take longer. It proposes WISE, a training‑free method that uses full attention in early steps and then reuses the discovered sparse working set for later steps, preserving performance on multi‑hop QA tasks. Experiments show that WISE maintains quality up to 2K context, offers measurable speedups, and highlights the importance of recurrent discovery of attention support.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

Related stories

arXiv AI
Oct 19

Prism Attention: Proposal-Refined Index Sharing Mechanism for Efficient LLMs Inference

Prism Attention introduces a proposal‑refined sparse attention framework that replaces static cross‑layer index reuse with a coarse‑to‑fine strategy, allowing layer‑wise adaptability while maintaining efficiency. The method leverages the observation that high‑attention tokens remain largely stable across layers, with changes concentrated near the boundary of previously selected regions. Experiments demonstrate state‑of‑the‑art performance on reasoning and long‑context benchmarks, achieving a 3.4× decoding speedup over FlashAttention at 128K context length.

By Zhenxu Tian, Kebin Liu, Zhengwu Yang, Yi Su, Qingqing Dang, Yanlin Sha, Kaipeng Deng, Dianhai Yu, Yanjun Ma, Juntao Li, Min Zhang
llmsefficiencybenchmarks
More like this →
arXiv AI
Oct 15

History Replays Itself: Unleashing Long-Context Capability in Linear Attention

arXiv:2610.15715v1 Announce Type: new Abstract: Linear attention promises the efficiency needed for long-context language modelling, yet its fixed-size recurrent state must simultaneously support ong...

By Li Nanbo, Firas Laakom, Yilan Zhang, Piotr Pi\k{e}kos, Yimeng Chen, Liangyu Wang, Yifan Xu, Mateusz Ostaszewski, Dandan Guo, Wenyi Wang, Yanning Dai, Yuhui Wang, J\"{u}rgen Schmidhuber
llmsragefficiency
More like this →
arXiv AI
Nov 6

Depth-Recurrent Attention Mixture (Dreamer): Giving Latent Reasoning the Attention it Deserves

The paper introduces Dreamer, a depth‑recurrent attention mixture framework that combines sequence attention, depth attention, and sparse expert attention to overcome the hidden‑size bottleneck in latent reasoning models. Dreamer decouples scaling dimensions, allowing efficient depth‑recurrent scaling, and achieves state‑of‑the‑art performance on language reasoning benchmarks while using 2 to 8× fewer training tokens than FLOP‑, parameter‑, and memory‑matched baselines. It also demonstrates significantly greater expert selection diversity, up to 11× larger than current MoE models.

By Jonas Knupp, Jan Hendrik Metzen, Jeremias Bohn, Georg Groh, Kristian Kersting
benchmarks
More like this →