Hugging Face Trending Papers
Sparse Attention Is Matrix Approximation, Not Choosing from a Bag of Values
Read the original on Hugging Face Trending Papers →The Flow has not summarised this story yet — read it at Hugging Face Trending Papers.
Related stories
arXiv Computation and Language
Oct 9
Sparse Attention Is Matrix Approximation, Not Choosing from a Bag of Values
The paper argues that sparse attention should be viewed as a matrix approximation problem rather than merely selecting large scalar entries from the attention matrix. It introduces Matrix Approximation Sparse Attention (MASA), a method that replaces raw attention-mass ranking with a closed‑form score reflecting the reduction in matrix‑product approximation error. MASA can be integrated into existing sparse attention frameworks and, according to extensive experiments, consistently improves accuracy across various methods, benchmarks, and LLM backbones.
arXiv AI
Jul 7
Hierarchical Sparse Attention Done Right: Toward Infinite Context Modeling
arXiv:2607. 02980v1 Announce Type: cross Abstract: Scaling modern large language models (LLMs) to long contexts is limited by the quadratic computation cost, and poor length extrapolation of dense attention.
arXiv AI
Jul 23
ELSAA: Efficient Low-Rank and Sparse Attention Approximation for Training Transformers
arXiv:2607. 20214v1 Announce Type: cross Abstract: The quadratic $N\times N$ attention score matrix remains a central obstacle to extending Transformers to longer input lengths.
arXiv Machine Learning
Jun 4
Customizing the Inductive Biases of Softmax Attention using Structured Matrices
arXiv:2509. 07963v2 Announce Type: replace Abstract: The core component of attention is the scoring function, which transforms the inputs into low-dimensional queries and keys and takes the dot product of each pair.
arXiv Computation and Language
Sep 14
SAS: Simple Attention Sparsification via End-to-End Optimization of Context Ranking
arXiv:2609.13141v1 Announce Type: new Abstract: Post-training attention sparsification reduces the quadratic cumulative attention cost of pretrained Transformers by selecting a small set of context u...
arXiv AI
Oct 7
More Value per Key: Asymmetric Sparse Attention for Faster LLM Decoding
arXiv:2610.04753v2 Announce Type: replace-cross Abstract: Autoregressive generation in Large Language Models (LLMs) is constrained by the memory and computational demands of attention mechanisms. Spa...