arXiv Machine Learning August 28, 2026 By Kasper Nordenram, Amelie Dittmann

ClusterAttention: A training-free speedup of bidirectional attention

Read the original on arXiv Machine Learning →

ClusterAttention is a training‑free technique that speeds up bidirectional attention by recursively clustering keys and queries into fixed‑size, power‑of‑two blocks, enabling block‑sparse attention to match dense attention latency on GPUs. The method derives error bounds for sparse attention, showing tighter clusters can reduce error when compensated via centroids, and demonstrates significant speedups—up to six‑fold on large tabular data and 1.8× on video generation—while preserving over 99% of dense accuracy.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

diffusion

arXiv AI

Jul 7

HyperVAttention: Efficient Sparse Attention with Spatio-Temporal Clustering for Video Diffusion

arXiv:2607. 03012v1 Announce Type: cross Abstract: Video Diffusion Transformers (VDiTs) have demonstrated significant capabilities in high-fidelity video generation.

By Dongyeun Lee, Amir Zandieh, Vahab Mirrokni, Junmo Kim, Insu Han

llms diffusion

Hugging Face Trending Papers

Aug 13

SCOPE: Subspace Clustering with Online Per-Head Top-K Estimation for Sparse Video Attention

Diffusion Transformers (DiTs) incur quadratic self-attention cost over spatiotemporal tokens. Existing training-free sparse attention methods often construct sparse masks from block-level or cluster-level proxy scores, which can obscure fine-grained differences among keys and miss high contribution keys under aggressive sparsity.

llms diffusion efficiency

arXiv Machine Learning

Jul 2

FSA: An Alternative Efficient Implementation of Native Sparse Attention Kernel

arXiv:2508. 18224v3 Announce Type: replace-cross Abstract: Recent advances in sparse attention mechanisms have demonstrated strong potential for reducing the computational cost of long-context training and inference in large language models (LLMs).

By Ran Yan, Youhe Jiang, Zhuoming Chen, Haohui Mai, Beidi Chen, Binhang Yuan

llms benchmarks

arXiv AI

Aug 20

Partition the Support, Reconstruct the Residual: Training-Free Sparse Attention for Video Generation and World Models

The paper introduces SparsePR, a training‑free block‑sparse attention method for video transformers that partitions query‑key responses and reconstructs the residual via probe‑fitted affine corrections. By pairing sampled‑query key responses into K/V groups and using centroids to guide shared routing, SparsePR reduces attention‑reconstruction error across diverse video generation and world‑model tasks. Experiments show consistent error reductions, with probe fitting contributing most of the improvement, while maintaining generation quality at 22.0–26.0% executed‑pair density and delivering 1.48×–2.61× speedups.

By Pardis Taghavi, Reza Langari, Gaurav Pandey

llms diffusion

arXiv AI

Jul 16

EMAGN: Efficient Multi-Attention Graph Network via Learned Clustering for Scalable Traffic Forecasting

arXiv:2607. 13241v1 Announce Type: cross Abstract: Traffic forecasting is highly challenging due to complex and nonlinear spatial and temporal dependencies.

By Mingxing Xu, Rakesh Chowdary Machineni, Ke Liu, Xi Cheng, Chengqi Lu, Xin Hu, Lyuhao Chen, Xiangyu Li, Junwei You, Oliver Gao

benchmarks

arXiv AI

Aug 13

LoSA: Near-Lossless Sparse Attention for Training-Free Video Diffusion Acceleration

arXiv:2608. 12032v1 Announce Type: cross Abstract: Video diffusion transformers are costly to sample: every denoising step applies self-attention over a long 3D token sequence, a quadratic cost that dominates as resolution and duration grow.

By Enhuai Liu, Yunke Wang, Yutong Wang, Changming Sun, Chang Xu

llms diffusion efficiency

One email a morning, machine-written

Related stories

HyperVAttention: Efficient Sparse Attention with Spatio-Temporal Clustering for Video Diffusion

SCOPE: Subspace Clustering with Online Per-Head Top-K Estimation for Sparse Video Attention

FSA: An Alternative Efficient Implementation of Native Sparse Attention Kernel

Partition the Support, Reconstruct the Residual: Training-Free Sparse Attention for Video Generation and World Models

EMAGN: Efficient Multi-Attention Graph Network via Learned Clustering for Scalable Traffic Forecasting

LoSA: Near-Lossless Sparse Attention for Training-Free Video Diffusion Acceleration