arXiv:2608.30310v1 Announce Type: cross
Abstract: Hybrid large language models interleave full-attention layers with linear-attention layers to reduce the cost of long-context inference. This structu...
By Yirui Liu, Ruoling Qi, Xuaner Wu, Penghang Liu, Jian Chen
The paper presents a method for translating key‑value (KV) caches between hybrid‑attention language models, enabling a model to continue a conversation without re‑reading the entire history. It extends prior work (KV‑Lingo) to handle grouped‑query attention, compressed MLA latents, sparse‑attention indexer keys, and fixed‑size linear‑attention states, providing capture‑and‑inject points and mechanisms that preserve native quality. Experiments on 16–48 B hybrid pairs show near‑native performance and significant speedups, while also demonstrating effective long‑range recall and transfer to sparse‑attention targets.
arXiv:2610.15715v1 Announce Type: new
Abstract: Linear attention promises the efficiency needed for long-context language modelling, yet its fixed-size recurrent state must simultaneously support ong...
By Li Nanbo, Firas Laakom, Yilan Zhang, Piotr Pi\k{e}kos, Yimeng Chen, Liangyu Wang, Yifan Xu, Mateusz Ostaszewski, Dandan Guo, Wenyi Wang, Yanning Dai, Yuhui Wang, J\"{u}rgen Schmidhuber
arXiv:2607. 27539v2 Announce Type: replace Abstract: Exact deletion from persistent language-model memory depends on whether a record's effect remains addressable after later computation.
By Vishwajith Ramesh
arXiv:2608.30386v1 Announce Type: cross
Abstract: Hybrid linear-attention architectures have recently scaled to large open-weight models, offering quality competitive with full attention while substa...
By Yanqi Yu, Pingwei Sun, Jianchao Tan, Tao Zhang, Yuchen Xie, Xunliang Cai, Yao Liu
arXiv:2609.23900v1 Announce Type: new
Abstract: Tree speculative decoding verifies multiple candidate continuations in one target forward pass. For attention-only transformers, the verifier mainly ne...
By Zhiyuan Ma
Hybrid language models combine attention with a fixed-size recurrent state, yet the distinct roles of each component are not well understood. The authors introduce two cache-level interventions—split-prefill and state-swap—to isolate the contributions of the KV cache (attention) and the recurrent state. Experiments on Qwen3.5 and Falcon-H1 show that exact retrieval depends almost entirely on attention, while output language and persona rely mainly on recurrence, with the state-swap intervention confirming that answers derive from the KV side and language from the recurrent side.
By Kirill Afendulev, Alexey Dontsov, Elena Tutubalina, Anton Korznikov
The paper introduces Hybrid State Transfer (HST), a method for transferring not only full‑attention key/value caches but also recurrent (REC) and convolution (Conv) states between different large language models during long conversations. HST preserves the order of Conv histories and uses structured REC maps, calibrating adapters with the target model’s outputs to maintain continuity. Experiments on five benchmarks show that Qwen3.5 models achieve high task retention (up to 94%) and achieve over 21× speedups on synthetic 16K inputs when handoff is performed.
By Jiacheng Cai, Tao Zhang, Siqi Liang, Yun Dai, Ziqian Zeng
arXiv:2607. 21000v1 Announce Type: new Abstract: Long-sequence memory tracking places two opposing demands on a recurrent state: near-lossless retention of stored bindings over long horizons, and active overwriting of stale ones.
By Hyuk Lim, Seunghyun Yoon
The paper introduces SEED, a method that reduces the computational burden of hybrid language models by making both the prefill and decode phases shallow. SEED exploits two key properties of trained hybrids: (1) deep layer memories can be written efficiently from shallower layers, and (2) recurrent states contain a dominant rank‑one component that can be stored compactly. By freezing the original weights and compressing recurrent states into a sink plus low‑rank factors, SEED achieves near‑lossless performance on large models while boosting per‑GPU throughput by 1.2–1.4× on NVIDIA GB300 GPUs.
By Mingyuan Ma, Yangmin Li, Harris Nover, Zheyu Fu, Po-Han Huang, Alex Settle, Ashwin Nanjappa
arXiv:2606. 16093v1 Announce Type: cross Abstract: Modeling long-range dependencies remains a central challenge in natural language processing.
By Kuzey Torlak, H\"useyin Arda Arslan, An{\i}l Dervi\c{s}o\u{g}lu, Beyza Nur Deniz, Onur Boyar
arXiv:2603. 06642v2 Announce Type: replace-cross Abstract: Test-Time Training (TTT) language models replace the KV-cache with fast weights updated during inference, achieving O(1) memory but suffering catastrophic failure on exact-recall tasks.
By Swamynathan V P