arXiv Machine Learning
Approaching I/O-optimality for Approximate Attention
Read the original on arXiv Machine Learning →The Flow has not summarised this story yet — read it at arXiv Machine Learning.
Related stories
arXiv Machine Learning
Sep 30
Efficiently Approximating Attention Is Hard
arXiv:2609.37261v1 Announce Type: new Abstract: Softmax attention is ubiquitous in modern machine learning, but its quadratic scaling with sequence length makes it costly. To reduce this cost, attent...
arXiv Machine Learning
Oct 8
Attention via Black-Box Vector Search
arXiv:2610.10135v1 Announce Type: cross Abstract: Sparse attention mechanisms estimate attention over $n$ tokens using a small subset of keys. Many existing approaches use maximum inner product searc...
arXiv Machine Learning
1d ago
Softmax Attention with Any Temperature in Fixed Dimension: Matching the Best Known Maximum-Inner-Product Bounds
arXiv:2611.06055v1 Announce Type: cross Abstract: Attention is central to modern AI models for language, vision, and scientific modeling. However, direct evaluation compares every query with every ke...
arXiv Machine Learning
Sep 22
FlashBoB: I/O-Efficient Exact Backward-over-Backward for Softmax Attention
FlashBoB introduces an I/O‑efficient algorithm for exact backward‑over‑backward (BoB) in softmax attention, enabling precise second‑order differentiation without large intermediate tensors. By exploiting a hierarchical affine structure, the method confines computation to on‑chip tiles and limits off‑chip memory traffic, achieving θ(N² d²/M) HBM usage. Experiments show FlashBoB scales to sequence lengths of 262K on a single A100 GPU, outperforming prior exact baselines and FlashBack by up to 6.3×.
arXiv Machine Learning
Sep 30
Block Sparse Flash Attention
Block Sparse Flash Attention (BSFA) is a drop‑in replacement for FlashAttention that speeds up long‑context inference by pruning about 50% of computation and memory transfers. It selects the top‑k most important value blocks for each query using exact query‑key similarities and calibrated per‑layer, per‑head thresholds, requiring only a one‑time training‑free calibration. On Llama‑3.1‑8B, BSFA delivers up to 1.13× speedup on LongBench with a 1.1% accuracy drop and up to 1.24× on Needle‑in‑a‑Haystack retrieval with a 1% drop, while the attention kernel itself accelerates by up to 1.38×.
arXiv Computation and Language
Oct 9
Sparse Attention Is Matrix Approximation, Not Choosing from a Bag of Values
The paper argues that sparse attention should be viewed as a matrix approximation problem rather than merely selecting large scalar entries from the attention matrix. It introduces Matrix Approximation Sparse Attention (MASA), a method that replaces raw attention-mass ranking with a closed‑form score reflecting the reduction in matrix‑product approximation error. MASA can be integrated into existing sparse attention frameworks and, according to extensive experiments, consistently improves accuracy across various methods, benchmarks, and LLM backbones.