arXiv:2611. 07232v1 Announce Type: cross Abstract: At context lengths of about 1M tokens, dense attention is costly in prefill and decoding, and training typically relies on long documents, which become scarcer as the target length increases.
arXiv:2607. 27692v1 Announce Type: cross Abstract: Top-$K$ sparse attention reduces the cost of Softmax and value aggregation by attending to only a small subset of key--value (KV) entries.
HISA: Efficient Hierarchical Indexing for Fine-Grained Sparse Attention proposes a two-stage hierarchical indexer that replaces the flat token scan used in token-level sparse attention mechanisms like DeepSeek Sparse Attention. The method first performs block-level coarse filtering to discard irrelevant regions, then applies the original token-level indexer only within the retained candidate blocks, preserving the same top-sparse pattern for downstream attention. Benchmarks show HISA achieves significant speedups at 64K context and matches the quality of DeepSeek-V3.2 and GLM-5 without additional training.
arXiv:2610.42147v1 Announce Type: new Abstract: Agentic workflows are increasingly shifting towards long-horizon tasks, such as open-ended scientific work, where models benefit from long context wind...
The paper introduces Stratified Sparse Attention (SSA), a framework that blends deterministic top‑k retrieval with score‑guided sampling to reduce the gap between sparse and dense attention in large language models. SSA attends to high‑scoring tokens deterministically while sampling the remaining attention mass using proxy scores, thereby recovering more of the attention distribution. Experiments on benchmarks such as RULER, LongBench, LOFT, and AIME show that SSA outperforms existing sparse methods across models ranging from 3B to 27B parameters, achieving significant accuracy gains and up to 2.96× decode speedup with optimized CUDA kernels.
arXiv:2604. 20920v2 Announce Type: replace Abstract: Sparse attention can reduce the cost of long-context inference, but most variants introduce new architectural components.
The paper introduces Zyphra Hierarchical Attention (ZHA), a subquadratic sparse attention mechanism that uses a tree-structured indexer to compress key-value pairs and retrieve a sparse subset of keys during query time. By iteratively deepening the tree, ZHA achieves sublinear routing and attention costs, resulting in overall subquadratic scaling for long-context processing. The method can be applied to existing pretrained models with minimal distillation training, maintains performance on diverse evaluation tasks, and remains robust even with very deep trees.
Joint Subspace Sparse Attention (JSSA) is a new method that reduces KV‑cache traffic in long‑context LLM decoding by selecting tokens at the token level while preserving query–key structure. It learns a shared low‑dimensional query–key subspace through a tractable spectral solution and implements a gather‑free indexed‑attention kernel that supports per‑KV‑head selections. Across various attention architectures, JSSA maintains near full‑KV accuracy, avoids generation‑length inflation, and can achieve up to 6.43× decoding throughput on a single GPU for 128K context, even at 700B‑parameter scale.
Sparse attention reduces the KV-cache traffic that dominates long-context LLM decoding, but practical methods largely operate at coarse granularity. Token-level selection offers finer control, yet pre...
arXiv:2606. 13392v1 Announce Type: new Abstract: Ultra-long-context capability is becoming indispensable for frontier LLMs: agentic workflows, repository-scale code reasoning, and persistent memory all require the model to jointly attend over hundreds of thousands to millions of tokens, yet the quadratic cost of softmax attention makes this untenable at deployment scale.
arXiv:2609. 31093v1 Announce Type: new Abstract: Scaling language models to long contexts is limited by the quadratic cost of self-attention.
arXiv:2610.14754v1 Announce Type: new Abstract: Hybrid language models interleave Gated DeltaNet (GDN) layers, whose recurrent memory has a fixed size, with full-attention layers, whose key-value (KV...
arXiv:2605. 16928v2 Announce Type: replace-cross Abstract: Long-context inference in large language models is bottlenecked by the quadratic cost of full attention.