Sparse Attention Is Matrix Approximation, Not Choosing from a Bag of Values

The paper argues that sparse attention should be viewed as a matrix approximation problem rather than merely selecting large scalar entries from the attention matrix. It introduces Matrix Approximation Sparse Attention (MASA), a method that replaces raw attention-mass ranking with a closed‑form score reflecting the reduction in matrix‑product approximation error. MASA can be integrated into existing sparse attention frameworks and, according to extensive experiments, consistently improves accuracy across various methods, benchmarks, and LLM backbones.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computation and Language.

• llms
• benchmarks