arXiv AI
Latent Sparse Attention: Towards Subquadratic Ultra-Long Context Attention
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
Hugging Face Trending Papers
2d ago
AttentionIndex: Accelerating Learned Sparse Attention with a Causal Inverted-File Index
Learned sparse attention, used in production models such as DeepSeek-V4 and GLM-5, lets each query attend to only k keys chosen by a lightweight indexer, but the indexer still scores every key in the...
arXiv AI
Jun 9
Full Attention Strikes Back: Transferring Full Attention into Sparse within Hundred Training Steps
arXiv:2605. 16928v2 Announce Type: replace-cross Abstract: Long-context inference in large language models is bottlenecked by the quadratic cost of full attention.
arXiv AI
Jul 7
Hierarchical Sparse Attention Done Right: Toward Infinite Context Modeling
arXiv:2607. 02980v1 Announce Type: cross Abstract: Scaling modern large language models (LLMs) to long contexts is limited by the quadratic computation cost, and poor length extrapolation of dense attention.
arXiv AI
Oct 26
HyScale: Scaling Long-Context Training for Learned Sparse Attention
HyScale is a long‑context training system designed to accelerate learned sparse attention for million‑token contexts. It introduces HyTopK, which computes each query’s exact top‑k indices on the fly without a full score table, and selection replay, which reuses forward‑pass selections during recomputation. These techniques, combined with system‑wide optimizations, achieve a 2.07× speedup for GLM‑5.1 training on 1M‑token sequences compared to Megatron‑LM, and similar gains for DeepSeek‑V4‑Flash and Qwen3.8‑Flash‑Next on large GPU clusters.
arXiv Machine Learning
Sep 11
HISA: Efficient Hierarchical Indexing for Fine-Grained Sparse Attention
HISA: Efficient Hierarchical Indexing for Fine-Grained Sparse Attention proposes a two-stage hierarchical indexer that replaces the flat token scan used in token-level sparse attention mechanisms like DeepSeek Sparse Attention. The method first performs block-level coarse filtering to discard irrelevant regions, then applies the original token-level indexer only within the retained candidate blocks, preserving the same top-sparse pattern for downstream attention. Benchmarks show HISA achieves significant speedups at 64K context and matches the quality of DeepSeek-V3.2 and GLM-5 without additional training.
arXiv AI
Jul 23
LISA: Linear-Indexed Sparse Attention for Efficient Long-Context Reasoning
arXiv:2607. 19358v1 Announce Type: new Abstract: Recent advances in long chain-of-thought reasoning models such as DeepSeek-R1 have led to increasingly longer inference context lengths under the test-time scaling paradigm.