arXiv Computation and Language

What Attention Recalls and Recurrence Controls in Hybrid Language Models

Hybrid language models combine attention with a fixed-size recurrent state, yet the distinct roles of each component are not well understood. The authors introduce two cache-level interventions—split-prefill and state-swap—to isolate the contributions of the KV cache (attention) and the recurrent state. Experiments on Qwen3.5 and Falcon-H1 show that exact retrieval depends almost entirely on attention, while output language and persona rely mainly on recurrence, with the state-swap intervention confirming that answers derive from the KV side and language from the recurrent side.

arXiv AI
Sep 3

Language Models Can Control Their Own Attention

The paper introduces Declarative Attention (DA), a protocol that lets language models explicitly declare which parts of their context to focus on during generation. By partitioning decoding into full-context, region-specific, and recent-output-only modes, the inference engine can skip large portions of the KV cache, dramatically reducing attended tokens. Experiments on 15 long-context tasks with off-the-shelf models show significant savings (52.0% and 31.1% reductions) with only modest accuracy drops that diminish as model size increases.

By Namgyu Ho, Huzama Ahmad, Woosung Koh, Se-Young Yun, Tal Schuster, Cicero Nogueira dos Santos
Hugging Face Trending Papers
Nov 9

Translating KV Caches Between Hybrid-Attention Language Models: MLA Latents, Linear-Attention States and Sparse-Attention Indexers

The paper presents a method for translating key‑value (KV) caches between hybrid‑attention language models, enabling a model to continue a conversation without re‑reading the entire history. It extends prior work (KV‑Lingo) to handle grouped‑query attention, compressed MLA latents, sparse‑attention indexer keys, and fixed‑size linear‑attention states, providing capture‑and‑inject points and mechanisms that preserve native quality. Experiments on 16–48 B hybrid pairs show near‑native performance and significant speedups, while also demonstrating effective long‑range recall and transfer to sparse‑attention targets.

arXiv Computation and Language
Sep 23

LatentPort: Beyond KV Cache - Cross-Model Transfer of Recurrent Memory in Hybrid Language Models: A 4B-to-9B Hybrid-State Handoff Without Target Prefix Replay

The paper introduces LatentPort, a method that allows a language model to transfer its live memory to another model without requiring the receiver to reread the context. Experiments on a Qwen3.5 4B-to-9B sibling pair show that adding a Gated DeltaNet (GDN) persistent-state package reduces negative log‑likelihood by 0.747 nats/token and improves performance across 64 PG19 documents. The study also demonstrates that direct recurrent and convolution reuse outperforms learned GDN maps, and a 434,176‑parameter correction further narrows the performance gap to the native 9B model.

By Simon P. Villani
arXiv Machine Learning
Sep 24

Attention Routing Stabilizes Early: Working-Set Inference for Recurrent Language Models

The paper investigates how attention dynamics evolve across recurrent depth in language models, finding that attention support stabilizes early while hidden states and outputs take longer. It proposes WISE, a training‑free method that uses full attention in early steps and then reuses the discovered sparse working set for later steps, preserving performance on multi‑hop QA tasks. Experiments show that WISE maintains quality up to 2K context, offers measurable speedups, and highlights the importance of recurrent discovery of attention support.

By Ke Wan, Chen Chen
arXiv AI
Oct 15

History Replays Itself: Unleashing Long-Context Capability in Linear Attention

arXiv:2610.15715v1 Announce Type: new Abstract: Linear attention promises the efficiency needed for long-context language modelling, yet its fixed-size recurrent state must simultaneously support ong...

By Li Nanbo, Firas Laakom, Yilan Zhang, Piotr Pi\k{e}kos, Yimeng Chen, Liangyu Wang, Yifan Xu, Mateusz Ostaszewski, Dandan Guo, Wenyi Wang, Yanning Dai, Yuhui Wang, J\"{u}rgen Schmidhuber
arXiv Computation and Language
Nov 6

Beyond K/V Caches: Cross-Model State Handoff in Hybrid Linear-Attention LLMs

The paper introduces Hybrid State Transfer (HST), a method for transferring not only full‑attention key/value caches but also recurrent (REC) and convolution (Conv) states between different large language models during long conversations. HST preserves the order of Conv histories and uses structured REC maps, calibrating adapters with the target model’s outputs to maintain continuity. Experiments on five benchmarks show that Qwen3.5 models achieve high task retention (up to 94%) and achieve over 21× speedups on synthetic 16K inputs when handoff is performed.

By Jiacheng Cai, Tao Zhang, Siqi Liang, Yun Dai, Ziqian Zeng
arXiv Computation and Language
Sep 18

On-Demand Attention: Language Models Know When to Recall

The paper introduces On‑Demand Attention (ODA), a decoding strategy that lets pretrained language models decide when to use global attention based on a lightweight recall head. ODA keeps the original model weights unchanged, only training the recall head, and can be implemented with GPU‑side conditional execution to reduce global reads. Experiments on Qwen, Gemma, and hybrid‑attention models show that ODA largely recovers performance lost by local attention while cutting the number of global attention operations.

By Haibo Feng, Ruiqi Liang, Hanyang Peng, Shiqi Yu
arXiv Machine Learning
Sep 18

dQwen3.5: Hybrid-Attention Diffusion Language Models

The paper introduces dQwen3.5, a family of diffusion language models derived from the hybrid-attention architecture of Qwen3.5 at 0.8B, 2B, 4B, and 9B parameters. It demonstrates that adapting a hybrid backbone—combining attention and RNN layers—can be more efficient than full-attention models, reaching a target training loss in roughly half the tokens. Across scales, dQwen3.5 exhibits full-attention-like behavior in any-order decoding and strong performance with parallel decoding.

By Anton Xue, Litu Rout, Aditya Akella, Adam Klivans, Sujay Sanghavi, Sanjay Shakkottai