The paper introduces Memory Attention (MA), a new attention mechanism for language models that replaces the traditional value projection with token-indexed memory combined with contextual keys. MA generates values by merging layer‑specific token memory with contextual information, allowing normalization to be folded into memory tables and reducing value construction to a simple lookup and addition. Experiments show that, with matched training token budgets and additional memory parameters, MA improves language modeling performance and average downstream task results across various attention configurations.
Language models typically construct attention values from contextual hidden states, even when some of their content may be reusable across contexts. We investigate whether token-indexed memory can rep...
A lot of prior work addressed key-value (KV) cache selection and compression by sparse attention to enable long-context inference for transformer language models without excessive hardware budgets. We provide a new method for fine-tuning models with sparse attention.
arXiv:2608. 19920v1 Announce Type: new Abstract: A lot of prior work addressed key-value (KV) cache selection and compression by sparse attention to enable long-context inference for transformer language models without excessive hardware budgets.
arXiv:2609.39661v1 Announce Type: new Abstract: Self-attention gives LLMs fine-grained, query-dependent access to context, but dense token interactions incur quadratic prefill cost and a key--value c...
arXiv:2610.07940v1 Announce Type: cross Abstract: Looped language models apply the same stack of layers T times to each token, which deepens the model without adding parameters but multiplies its key...
Prism Attention introduces a proposal‑refined sparse attention framework that replaces static cross‑layer index reuse with a coarse‑to‑fine strategy, allowing layer‑wise adaptability while maintaining efficiency. The method leverages the observation that high‑attention tokens remain largely stable across layers, with changes concentrated near the boundary of previously selected regions. Experiments demonstrate state‑of‑the‑art performance on reasoning and long‑context benchmarks, achieving a 3.4× decoding speedup over FlashAttention at 128K context length.
The paper introduces On‑Demand Attention (ODA), a local‑first decoding strategy that predicts when a pretrained language model would benefit from global attention. By training only a lightweight recall head, ODA selectively triggers global attention during generation, keeping pretrained weights unchanged and preserving the full key‑value cache for future recall. Experiments on Qwen, Gemma, and hybrid‑attention models show that ODA largely recovers performance lost with local attention while significantly cutting global reads, enabling faster long‑context inference.
The paper introduces On‑Demand Attention (ODA), a decoding strategy that lets pretrained language models decide when to use global attention based on a lightweight recall head. ODA keeps the original model weights unchanged, only training the recall head, and can be implemented with GPU‑side conditional execution to reduce global reads. Experiments on Qwen, Gemma, and hybrid‑attention models show that ODA largely recovers performance lost by local attention while cutting the number of global attention operations.
arXiv:2610.19210v1 Announce Type: new Abstract: Transformers are efficient to pre-train at scale because attention permits parallel computation across sequence positions, but long-context inference r...
The paper introduces NAMOH, a native sparse attention mechanism that activates only a subset of heads per token, allowing each head to attend to a limited subsequence of tokens. By scaling the number of heads while keeping the active heads per token fixed, the method shortens head histories and reduces key‑value access without increasing overall storage. Experiments demonstrate that NAMOH can outperform fully activated models with the same parameter count and enable more efficient long‑context inference than smaller dense models.
The paper introduces Declarative Attention (DA), a protocol that lets language models explicitly declare which parts of their context to focus on during generation. By partitioning decoding into full-context, region-specific, and recent-output-only modes, the inference engine can skip large portions of the KV cache, dramatically reducing attended tokens. Experiments on 15 long-context tasks with off-the-shelf models show significant savings (52.0% and 31.1% reductions) with only modest accuracy drops that diminish as model size increases.
arXiv:2606. 18587v1 Announce Type: cross Abstract: Decoder-only Transformers compute attention over the KV cache of preceding tokens.