arXiv Machine Learning
Jun 18
Hierarchical Attention via Domain Decomposition
arXiv:2606. 18525v1 Announce Type: new Abstract: We propose a hierarchical attention mechanism based on two-level overlapping Schwarz domain decomposition.
Overlapping Schwarz Attention: Hierarchical Attention via Domain Decomposition
arXiv:2606. 18525v2 Announce Type: replace Abstract: We propose a hierarchical attention mechanism based on two-level overlapping Schwarz domain decomposition.
Related stories
arXiv AI
Jul 23
ELSAA: Efficient Low-Rank and Sparse Attention Approximation for Training Transformers
arXiv:2607. 20214v1 Announce Type: cross Abstract: The quadratic $N\times N$ attention score matrix remains a central obstacle to extending Transformers to longer input lengths.
arXiv Machine Learning
Jul 15
Semidirect Fourier Delta Attention: Phase-Controlled Delta Memory with Constructive Chunk-WY Kernels
arXiv:2607. 11897v1 Announce Type: new Abstract: Linear attention replaces softmax attention's growing KV cache with a fixed recurrent state, but this compression limits exact state tracking and long-context memory.
arXiv Machine Learning
Jul 1
Direction-Magnitude Decomposition for Low-Rank Matrix Optimization: Faster Convergence and Saddle-to-saddle Dynamics
arXiv:2606. 31390v1 Announce Type: cross Abstract: Low-rank matrix optimization is often carried out via the Burer-Monteiro (BM) formulation, but choosing the factorization rank $r$ is delicate and can substantially slow optimization.
Hugging Face Trending Papers
2d ago
SQuad: Sub-Quadratic Attention Distillation for Efficient Video Generation
Video Diffusion Transformers (DiTs) spend most of their compute inside the Self-Attention operation, whose cost grows quadratically, $\mathcal{O}(n^2)$, with the number of latent tokens $n$. For the task of video generation, the token count is large, so this term dominates runtime and memory, and thereby caps the resolution and duration we can generate.
arXiv Machine Learning
Jun 4
Customizing the Inductive Biases of Softmax Attention using Structured Matrices
arXiv:2509. 07963v2 Announce Type: replace Abstract: The core component of attention is the scoring function, which transforms the inputs into low-dimensional queries and keys and takes the dot product of each pair.
arXiv Machine Learning
Jun 5
When Attention Beats Fourier: Multi-Scale Transformers for PDE Solving on Irregular Domains
arXiv:2605. 08318v2 Announce Type: replace Abstract: We study the problem of \emph{architecture selection} for deep learning models trained to solve partial differential equations (PDEs), asking when transformer-based architectures with learned attention outperform Fourier-domain neural operators.
arXiv AI
Jun 10
Blurry Window Attention
arXiv:2606. 09862v1 Announce Type: cross Abstract: The Softmax Attention operation in Transformer language models has a quadratic complexity in the sequence length and a growing state size in the form of KV cache, which becomes a bottleneck in long context scenarios.
arXiv Machine Learning
Jul 2
FRAME: Learning the Adaptation Domain with a Mixture of Fractional-Fourier Experts
arXiv:2607. 00162v1 Announce Type: new Abstract: Parameter-efficient fine-tuning (PEFT) reparameterizes weight updates in a fixed basis: low-rank adapters operate in the spatial domain, while a recent line of spectral methods operates in a fixed Fourier domain.
arXiv Machine Learning
Jun 5
When Good Enough Is Optimal: Multiplication-Only Matrix Inversion Approximation for Quantized Gated DeltaNet
arXiv:2606. 06034v1 Announce Type: new Abstract: Matrix inversion in chunk-wise parallel linear attention is a major bottleneck for long-context modeling, particularly on NPUs, where forward-substitution-based methods exhibit limited parallelism and poor hardware utilization.
arXiv AI
Aug 5
$\pi$-Attention: Online Efficient Sparse Transformers for Long-Context Modeling
arXiv:2511. 10696v3 Announce Type: replace-cross Abstract: Sparse attention is crucial in long-context Transformers, which restricts each token to a limited neighborhood and thereby reduces the quadratic cost of full self-attention.
arXiv Machine Learning
Aug 5
Provably Learning Multi-Head Attention with Queries
arXiv:2608. 03294v1 Announce Type: new Abstract: We study the problem of learning multi-head softmax attention from black-box input-output access.