arXiv:2608.30310v1 Announce Type: cross
Abstract: Hybrid large language models interleave full-attention layers with linear-attention layers to reduce the cost of long-context inference. This structu...
By Yirui Liu, Ruoling Qi, Xuaner Wu, Penghang Liu, Jian Chen
arXiv:2610.14754v1 Announce Type: new
Abstract: Hybrid language models interleave Gated DeltaNet (GDN) layers, whose recurrent memory has a fixed size, with full-attention layers, whose key-value (KV...
By Junjie Li, Jiong Lou, Jie Li
The paper investigates how block‑diffusion language models can use a constant‑size cache to enable efficient parallel decoding. By employing sequence mixers that summarize completed blocks into a reusable state and a block‑causal training objective, the authors pretrain three 3B block‑diffusion denoisers (attention, Mamba, and hybrid) on 300 B tokens. The resulting state‑space cache remains O(1) in memory and latency regardless of context length, yielding significant speed‑up and memory savings compared to traditional attention‑based caches, especially at very long sequences.
By Vaibhav Singh, Pierre-Andr\'e No\"el, Torsten Scholak, Eugene Belilovsky, Oleksiy Ostapenko
arXiv:2607. 16213v1 Announce Type: new Abstract: Large Language Models (LLMs) generate text autoregressively, relying on a key-value (KV) cache whose memory footprint grows linearly with context length, creating a major bottleneck.
By Soumia Bouyahiaoui, Manel Kara laouar, Aicha Boutorh, Mohamed Hadj Ameur
The paper introduces LatentPort, a method that allows a language model to transfer its live memory to another model without requiring the receiver to reread the context. Experiments on a Qwen3.5 4B-to-9B sibling pair show that adding a Gated DeltaNet (GDN) persistent-state package reduces negative log‑likelihood by 0.747 nats/token and improves performance across 64 PG19 documents. The study also demonstrates that direct recurrent and convolution reuse outperforms learned GDN maps, and a 434,176‑parameter correction further narrows the performance gap to the native 9B model.
By Simon P. Villani
arXiv:2610.33025v1 Announce Type: cross
Abstract: Letting a language model take gradient steps on the text it is reading, known as dynamic evaluation, lowers its loss on the rest of the document. Its...
By Surya Saka
arXiv:2610.23290v1 Announce Type: new
Abstract: Diffusion-based multimodal large language models (dMLLMs) decode by iterative denoising with bidirectional attention, which precludes the standard auto...
By Qiyan Zhao, Xiaofeng Zhang, Xu-Yao Zhang, Shijie Han, Zhepeng Li, Shunzhi Zhu, Da-Han Wang
The paper introduces SEED, a method that reduces the computational burden of hybrid language models by making both the prefill and decode phases shallow. SEED exploits two key properties of trained hybrids: (1) deep layer memories can be written efficiently from shallower layers, and (2) recurrent states contain a dominant rank‑one component that can be stored compactly. By freezing the original weights and compressing recurrent states into a sink plus low‑rank factors, SEED achieves near‑lossless performance on large models while boosting per‑GPU throughput by 1.2–1.4× on NVIDIA GB300 GPUs.
By Mingyuan Ma, Yangmin Li, Harris Nover, Zheyu Fu, Po-Han Huang, Alex Settle, Ashwin Nanjappa
arXiv:2606. 28876v2 Announce Type: replace-cross Abstract: We study memory-managed long-context attention: explicit bounded memory with a learned query-independent writer, lifecycle control, query-aware reading, calibrated sparse fallback, and frozen-LLM generation from raw evidence.
By Junyi Zou, Avrova Donz
arXiv:2603. 06642v2 Announce Type: replace-cross Abstract: Test-Time Training (TTT) language models replace the KV-cache with fast weights updated during inference, achieving O(1) memory but suffering catastrophic failure on exact-recall tasks.
By Swamynathan V P
arXiv:2610.15715v1 Announce Type: new
Abstract: Linear attention promises the efficiency needed for long-context language modelling, yet its fixed-size recurrent state must simultaneously support ong...
By Li Nanbo, Firas Laakom, Yilan Zhang, Piotr Pi\k{e}kos, Yimeng Chen, Liangyu Wang, Yifan Xu, Mateusz Ostaszewski, Dandan Guo, Wenyi Wang, Yanning Dai, Yuhui Wang, J\"{u}rgen Schmidhuber
arXiv:2607. 27269v1 Announce Type: new Abstract: Multi-head latent attention (MLA) is increasingly important for long-context LLM inference because compact latent states replace the growing key-value (KV) cache and reduce decoding memory traffic.
By Weiye Shi, Fanxu Meng, Muhan Zhang