arXiv Machine Learning
Jun 4
SparDA: Sparse Decoupled Attention for Efficient Long-Context LLM Inference
arXiv:2606. 04511v1 Announce Type: cross Abstract: Sparse attention reduces compute and memory bandwidth for long-context LLM inference.
MiniMax Sparse Attention
arXiv:2606. 13392v1 Announce Type: new Abstract: Ultra-long-context capability is becoming indispensable for frontier LLMs: agentic workflows, repository-scale code reasoning, and persistent memory all require the model to jointly attend over hundreds of thousands to millions of tokens, yet the quadratic cost of softmax attention makes this untenable at deployment scale.
Related stories
arXiv Machine Learning
Jul 2
FSA: An Alternative Efficient Implementation of Native Sparse Attention Kernel
arXiv:2508. 18224v3 Announce Type: replace-cross Abstract: Recent advances in sparse attention mechanisms have demonstrated strong potential for reducing the computational cost of long-context training and inference in large language models (LLMs).
arXiv AI
Jul 22
GQLA: Group-Query Latent Attention for Hardware-Adaptive Large Language Model Decoding
arXiv:2605. 15250v3 Announce Type: replace-cross Abstract: Multi-head Latent Attention (MLA), the attention used in DeepSeek-V2/V3, jointly compresses keys and values into a low-rank latent and matches the H100 roofline almost perfectly.
arXiv Machine Learning
Sep 11
HISA: Efficient Hierarchical Indexing for Fine-Grained Sparse Attention
HISA: Efficient Hierarchical Indexing for Fine-Grained Sparse Attention proposes a two-stage hierarchical indexer that replaces the flat token scan used in token-level sparse attention mechanisms like DeepSeek Sparse Attention. The method first performs block-level coarse filtering to discard irrelevant regions, then applies the original token-level indexer only within the retained candidate blocks, preserving the same top-sparse pattern for downstream attention. Benchmarks show HISA achieves significant speedups at 64K context and matches the quality of DeepSeek-V3.2 and GLM-5 without additional training.
arXiv AI
Jun 9
FlashMemory-DeepSeek-V4: Lightning Index Ultra-Long Context via Lookahead Sparse Attention
arXiv:2606. 09079v1 Announce Type: cross Abstract: Conventional LLMs keep the full KV cache loaded during decoding, causing a severe GPU memory bottleneck for ultra-long context serving.
arXiv Machine Learning
4d ago
Block Sparse Flash Attention
Block Sparse Flash Attention (BSFA) is a drop‑in replacement for FlashAttention that speeds up long‑context inference by pruning about 50% of computation and memory transfers. It selects the top‑k most important value blocks for each query using exact query‑key similarities and calibrated per‑layer, per‑head thresholds, requiring only a one‑time training‑free calibration. On Llama‑3.1‑8B, BSFA delivers up to 1.13× speedup on LongBench with a 1.1% accuracy drop and up to 1.24× on Needle‑in‑a‑Haystack retrieval with a 1% drop, while the attention kernel itself accelerates by up to 1.38×.
arXiv Machine Learning
Sep 22
FlashBoB: I/O-Efficient Exact Backward-over-Backward for Softmax Attention
FlashBoB introduces an I/O‑efficient algorithm for exact backward‑over‑backward (BoB) in softmax attention, enabling precise second‑order differentiation without large intermediate tensors. By exploiting a hierarchical affine structure, the method confines computation to on‑chip tiles and limits off‑chip memory traffic, achieving θ(N² d²/M) HBM usage. Experiments show FlashBoB scales to sequence lengths of 262K on a single A100 GPU, outperforming prior exact baselines and FlashBack by up to 6.3×.
arXiv Machine Learning
Sep 21
Elastic Threshold Attention: Learned Contextual Sparsity for Long-Context Decoding
Elastic Threshold Attention (ETA) is a trainable attention mechanism that dynamically predicts contextual thresholds from query representations, enabling selective pruning of KV cache tokens during long‑context decoding. By multiplicatively suppressing sub‑threshold logits during training, ETA avoids representation collapse and eliminates localized attention sinks, allowing a 1.45B model to match dense attention performance at roughly 85% training sparsity and 38% active decode density. At inference, a custom Triton kernel achieves up to 2.5× faster decoding on sequences up to 512K tokens, and an offline calibration step can further reduce compute by 27% by freezing per‑head thresholds.
arXiv AI
Jul 1
Hierarchical Global Attention (HGA)
arXiv:2606. 30709v1 Announce Type: cross Abstract: Hierarchical Global Attention (HGA) is a drop-in replacement for dense causal attention in pretrained long-context transformers.
arXiv Machine Learning
Jul 13
COBS: Cumulant Order Block Sparse Attention
arXiv:2607. 09052v1 Announce Type: new Abstract: Block sparse attention is a hardware friendly way to alleviate the key-value (KV) cache read bottleneck in large language models (LLMs).
arXiv AI
Sep 21
RBS-Attention: Radius-Bounded Sparse Prefill for Long-Context Large Language Models
RBS-Attention introduces a training‑free, radius‑bounded sparse prefill strategy for long‑context large language models, addressing the mean dilution problem where a block centroid can miss highly relevant tokens. The method employs two complementary selection branches: a centroid base branch that captures average relevance and a rescue branch that uses the maximum key‑block radius to flag under‑estimated blocks. Experiments on Qwen3 models demonstrate significant speedups—over 20× in standalone prefill‑attention and nearly 6× in end‑to‑first‑token time—while maintaining competitive accuracy compared to dense attention.
arXiv Computation and Language
Sep 18
DeepSeek-V4.1-Flash: Pushing the Limits of KV Cache Compression
DeepSeek‑V4.1‑Flash is a multimodal Mixture‑of‑Experts model with 552 B backbone parameters that supports contexts of up to one million tokens. It uses a Causal Encoder‑Decoder architecture that activates 16 B parameters per token during decoding but only 8 B during prefill, improving cost efficiency for agentic workloads. The model introduces cross‑layer KV cache reuse via Compressed Sparse Attention 2 and FP4 KV caching, reducing its global KV cache footprint to 890 bytes per token and its persistent footprint to roughly one‑eighth of the previous version, while still delivering superior performance after pretraining on a 45 T‑token multimodal corpus.