arXiv Computer Vision
RoLA: Rotary-Positioned Low-Rank Linear Attention for Efficient Diffusion Transformers
Read the original on arXiv Computer Vision →The Flow has not summarised this story yet — read it at arXiv Computer Vision.
Related stories
Hugging Face Trending Papers
Aug 13
SCOPE: Subspace Clustering with Online Per-Head Top-K Estimation for Sparse Video Attention
Diffusion Transformers (DiTs) incur quadratic self-attention cost over spatiotemporal tokens. Existing training-free sparse attention methods often construct sparse masks from block-level or cluster-level proxy scores, which can obscure fine-grained differences among keys and miss high contribution keys under aggressive sparsity.
arXiv Computer Vision
Sep 1
Fast and Accurate Image Restoration and Generation with Rank Enhanced Linear Attention
arXiv:2505.16157v3 Announce Type: replace Abstract: Transformer-based models have made remarkable progress in image restoration (IR) tasks. However, the quadratic complexity of self-attention in Tran...
arXiv AI
Jul 23
ELSAA: Efficient Low-Rank and Sparse Attention Approximation for Training Transformers
arXiv:2607. 20214v1 Announce Type: cross Abstract: The quadratic $N\times N$ attention score matrix remains a central obstacle to extending Transformers to longer input lengths.
arXiv AI
Aug 13
LoSA: Near-Lossless Sparse Attention for Training-Free Video Diffusion Acceleration
arXiv:2608. 12032v1 Announce Type: cross Abstract: Video diffusion transformers are costly to sample: every denoising step applies self-attention over a long 3D token sequence, a quadratic cost that dominates as resolution and duration grow.
arXiv Computer Vision
2d ago
VC-Attention: Value Smoothing and Softmax Casting for Low-bit Attention
arXiv:2609.15810v1 Announce Type: new Abstract: Diffusion Transformers deliver state-of-the-art video generation, but their long spatiotemporal sequences make attention the dominant deployment cost,...
arXiv Machine Learning
Jun 5
Exact Linear Attention
arXiv:2605. 18848v3 Announce Type: replace Abstract: This paper introduces Exact Linear Attention (ELA), a mechanism that achieves linear computational complexity for Transformer attention by exploiting the exact decomposition property of kernel functions, thereby eliminating approximation error.