The paper introduces Flow-based Spectral Kernel Attention (FSKA), a framework that learns nonstationary attention kernels using spectral density learning and normalizing flows. FSKA replaces traditional query–key projections with a simplified design that retains fixed orthogonal value projections, leading to a more parameter‑efficient architecture. Experiments on classification tasks demonstrate competitive or improved performance while reducing trainable projection parameters by roughly 40%–47% and preserving linear‑attention memory scaling.
arXiv:2606. 27748v1 Announce Type: cross Abstract: Transformer models rely on attention mechanism to capture long-range dependencies but suffer from quadratic complexity, limiting their scalability to long sequences.
arXiv:2605. 18848v3 Announce Type: replace Abstract: This paper introduces Exact Linear Attention (ELA), a mechanism that achieves linear computational complexity for Transformer attention by exploiting the exact decomposition property of kernel functions, thereby eliminating approximation error.
The paper introduces Cubit, a Transformer‑style architecture that replaces the standard attention mechanism with Kernel Ridge Regression (KRR). By interpreting attention as Nadaraya‑Watson regression, Cubit incorporates the closed‑form KRR solution, combining kernel‑based value aggregation with normalization via the inverse kernel matrix. The authors also propose a Limited‑Range Rescale (LRR) to stabilize training and report that Cubit shows improved long‑sequence modeling, with gains increasing as training sequence length grows.
arXiv:2606. 08327v1 Announce Type: cross Abstract: Standard transformers apply self-attention uniformly at every layer and token, regardless of whether the input requires dynamic cross-token interaction.
The paper introduces attention kernels that replace the exponential function in transformer softmax to better handle operator learning on probability measures with heavy-tailed (polynomial) distributions. Two new benchmarks with closed‑form targets are constructed to evaluate how different kernel growth rates and data preprocessing affect performance. The study finds that slower‑growing kernels prevent ensemble collapse on heavy‑tailed tasks, while softmax with symlog preprocessing only succeeds on a subset of problems, and that all kernels perform similarly on Gaussian data.
arXiv:2607. 07478v1 Announce Type: new Abstract: FFT-based spectral preprocessing of learned query-key (Q/K) projections substantially improves transformer attention on character-level language modelling.
arXiv:2610.14219v1 Announce Type: cross Abstract: Softmax attention in Transformers suffers from quadratic complexity in sequence length, making it impractical for long-context applications. Linear a...
arXiv:2603.02561v2 Announce Type: replace-cross Abstract: Attention mechanism remains the defining operator in Transformers since it provides expressive global credit assignment, yet its quadratic co...
arXiv:2609.22506v1 Announce Type: new Abstract: Vision Transformers allocate most parameters to multi-layer perceptrons (MLPs) for channel mixing, while token interactions usually rely on quadratic m...
The paper introduces SPD-MetaFormer, an attention‑free architecture for decoding brain signals from small datasets. It shows that existing attention‑based models on the SPD manifold (MAtt and GBWAtt) learn nearly uniform weights, and that replacing learned weights with uniform ones does not degrade performance. SPD-MetaFormer uses uniformly weighted Fréchet aggregation with a geodesic residual and a shared spectral feedforward map, achieving competitive results on three EEG benchmarks while maintaining SPD‑valued token states until classification.
The paper introduces QK Product Steering, a data‑free, training‑free method that edits the query‑key product in vision‑language models to reduce object hallucination. By suppressing a few dominant singular modes in selected middle layers and mapping the edited product back to query weights, the approach lowers hallucination rates without affecting inference cost. Experiments on three GQA‑based VLMs show a 4.0% average reduction in CHAIR$_s$, with the effect localized to symmetric mutual‑attention channels.
arXiv:2609.13141v1 Announce Type: new Abstract: Post-training attention sparsification reduces the quadratic cumulative attention cost of pretrained Transformers by selecting a small set of context u...