The paper introduces Zyphra Hierarchical Attention (ZHA), a subquadratic sparse attention mechanism that uses a tree-structured indexer to compress key-value pairs and retrieve a sparse subset of keys during query time. By iteratively deepening the tree, ZHA achieves sublinear routing and attention costs, resulting in overall subquadratic scaling for long-context processing. The method can be applied to existing pretrained models with minimal distillation training, maintains performance on diverse evaluation tasks, and remains robust even with very deep trees.
arXiv:2604. 20920v2 Announce Type: replace Abstract: Sparse attention can reduce the cost of long-context inference, but most variants introduce new architectural components.
arXiv:2607. 19358v1 Announce Type: new Abstract: Recent advances in long chain-of-thought reasoning models such as DeepSeek-R1 have led to increasingly longer inference context lengths under the test-time scaling paradigm.
arXiv:2609. 31093v1 Announce Type: new Abstract: Scaling language models to long contexts is limited by the quadratic cost of self-attention.
arXiv:2610.42147v1 Announce Type: new Abstract: Agentic workflows are increasingly shifting towards long-horizon tasks, such as open-ended scientific work, where models benefit from long context wind...
Prism Attention introduces a proposal‑refined sparse attention framework that replaces static cross‑layer index reuse with a coarse‑to‑fine strategy, allowing layer‑wise adaptability while maintaining efficiency. The method leverages the observation that high‑attention tokens remain largely stable across layers, with changes concentrated near the boundary of previously selected regions. Experiments demonstrate state‑of‑the‑art performance on reasoning and long‑context benchmarks, achieving a 3.4× decoding speedup over FlashAttention at 128K context length.
arXiv:2607. 02980v1 Announce Type: cross Abstract: Scaling modern large language models (LLMs) to long contexts is limited by the quadratic computation cost, and poor length extrapolation of dense attention.
HISA: Efficient Hierarchical Indexing for Fine-Grained Sparse Attention proposes a two-stage hierarchical indexer that replaces the flat token scan used in token-level sparse attention mechanisms like DeepSeek Sparse Attention. The method first performs block-level coarse filtering to discard irrelevant regions, then applies the original token-level indexer only within the retained candidate blocks, preserving the same top-sparse pattern for downstream attention. Benchmarks show HISA achieves significant speedups at 64K context and matches the quality of DeepSeek-V3.2 and GLM-5 without additional training.
arXiv:2604. 24432v2 Announce Type: replace-cross Abstract: Long-context ability, has become one of the most important iteration direction of next-generation Large Language Models, particularly in semantic understanding/reasoning, code agentic intelligence and recommendation system.
arXiv:2609.13141v1 Announce Type: new Abstract: Post-training attention sparsification reduces the quadratic cumulative attention cost of pretrained Transformers by selecting a small set of context u...
arXiv:2605. 16928v2 Announce Type: replace-cross Abstract: Long-context inference in large language models is bottlenecked by the quadratic cost of full attention.
arXiv:2608.28444v1 Announce Type: cross Abstract: Due to the nature of quadratic attention, Large Language Models (LLMs) consume a lot of memory and energy. Every new token costs more than the previo...
The paper introduces Stratified Sparse Attention (SSA), a framework that blends deterministic top‑k retrieval with score‑guided sampling to reduce the gap between sparse and dense attention in large language models. SSA attends to high‑scoring tokens deterministically while sampling the remaining attention mass using proxy scores, thereby recovering more of the attention distribution. Experiments on benchmarks such as RULER, LongBench, LOFT, and AIME show that SSA outperforms existing sparse methods across models ranging from 3B to 27B parameters, achieving significant accuracy gains and up to 2.96× decode speedup with optimized CUDA kernels.