arXiv AI
FreeSolo: Linear-time Sparse Attention with Zero-Shot Length Extrapolation
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv Machine Learning
Jun 16
LoLA: Low-Rank Linear Attention With Sparse Caching
arXiv:2505. 23666v3 Announce Type: replace-cross Abstract: The per-token cost of transformer inference scales with context length, preventing its application to lifelong in-context learning.
arXiv AI
Jul 23
LISA: Linear-Indexed Sparse Attention for Efficient Long-Context Reasoning
arXiv:2607. 19358v1 Announce Type: new Abstract: Recent advances in long chain-of-thought reasoning models such as DeepSeek-R1 have led to increasingly longer inference context lengths under the test-time scaling paradigm.
arXiv Computation and Language
Oct 1
Scaling Parameter and Context in Attention: Native Sparse Attention from Mixture-of-Head
The paper introduces NAMOH, a native sparse attention mechanism that activates only a subset of heads per token, allowing each head to attend to a limited subsequence of tokens. By scaling the number of heads while keeping the active heads per token fixed, the method shortens head histories and reduces key‑value access without increasing overall storage. Experiments demonstrate that NAMOH can outperform fully activated models with the same parameter count and enable more efficient long‑context inference than smaller dense models.
arXiv AI
Jul 7
Hierarchical Sparse Attention Done Right: Toward Infinite Context Modeling
arXiv:2607. 02980v1 Announce Type: cross Abstract: Scaling modern large language models (LLMs) to long contexts is limited by the quadratic computation cost, and poor length extrapolation of dense attention.
arXiv AI
Jun 9
Full Attention Strikes Back: Transferring Full Attention into Sparse within Hundred Training Steps
arXiv:2605. 16928v2 Announce Type: replace-cross Abstract: Long-context inference in large language models is bottlenecked by the quadratic cost of full attention.
Hugging Face Trending Papers
Aug 20
Learning how to Forget: Fine-tuning for Long-Context Sparse Attention
A lot of prior work addressed key-value (KV) cache selection and compression by sparse attention to enable long-context inference for transformer language models without excessive hardware budgets. We provide a new method for fine-tuning models with sparse attention.