arXiv AI October 12, 2026 By Weiqi Sun, Qianyue Wu, Wenjia Wang, Tianyang Hu

Not All Dimensions Are Equal: Outlier-Aware Attention Linearization with Grouped Quadratic Kernels

The paper introduces Outlier-Aware Attention Linearization (OAL), a method that improves linearized softmax attention by explicitly accounting for a few outlier dimensions that disproportionately affect query-key dot products. OAL builds on a second-order Taylor approximation, uses a Hadamard-product formulation to expose dimension-specific contributions, and groups dimensions with similar scales to share coefficients while treating outliers separately. Experiments on Qwen2.5-0.5B show an 8.9% reduction in frozen-attention output MSE, and fine-tuning with LoRA lowers WikiText-103 perplexity from 31.71 to 27.50 while boosting performance on PIQA, HellaSwag, and ARC-E.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

arXiv Machine Learning

Jun 5

Exact Linear Attention

arXiv:2605. 18848v3 Announce Type: replace Abstract: This paper introduces Exact Linear Attention (ELA), a mechanism that achieves linear computational complexity for Transformer attention by exploiting the exact decomposition property of kernel functions, thereby eliminating approximation error.

By Weinuo Ou

llms computer-vision reinforcement-learning efficiency safety

arXiv AI

Jul 23

ELSAA: Efficient Low-Rank and Sparse Attention Approximation for Training Transformers

arXiv:2607. 20214v1 Announce Type: cross Abstract: The quadratic $N\times N$ attention score matrix remains a central obstacle to extending Transformers to longer input lengths.

By Mahdi Heidari, Mohammad Mahdi Rahimi, Jaekyun Moon

llms efficiency

arXiv Machine Learning

Jun 4

Customizing the Inductive Biases of Softmax Attention using Structured Matrices

arXiv:2509. 07963v2 Announce Type: replace Abstract: The core component of attention is the scoring function, which transforms the inputs into low-dimensional queries and keys and takes the dot product of each pair.

By Yilun Kuang, Noah Amsel, Sanae Lotfi, Shikai Qiu, Andres Potapczynski, Andrew Gordon Wilson

safety

arXiv AI

Oct 15

Learnable Low-Rank Polynomial Sketch for Effective Linear Attention

arXiv:2610.14219v1 Announce Type: cross Abstract: Softmax attention in Transformers suffers from quadratic complexity in sequence length, making it impractical for long-context applications. Linear a...

By Zongyue Qin, Haoran Deng, Jason Cong, Yizhou Sun

llms benchmarks

arXiv Machine Learning

Jun 2

Don't Read Everything: A Curvature-Conditioned Query for Linear Attention

arXiv:2606. 01294v1 Announce Type: cross Abstract: Linear attention reduces the quadratic cost of softmax attention by maintaining a recurrent fast-weight state, but it consistently lags on in-context retrieval and long-context tasks.

By Dong Le, Thong Nguyen, Cong-Duy Nguyen, Anh Tuan Luu

arXiv Machine Learning

Jul 28

The Entropic Bound for Transformers: Why Static Rank Fails and Attention-Native Rank Recovers

arXiv:2607. 23050v1 Announce Type: new Abstract: Neural scaling laws describe how loss decreases as models, data, and compute grow, but they do not answer a prior question: for a fixed task, what is the minimum model capacity required to solve it?

By Byeong Hoon Yoon

llms safety

One email a morning, machine-written

Related stories

Exact Linear Attention

ELSAA: Efficient Low-Rank and Sparse Attention Approximation for Training Transformers

Customizing the Inductive Biases of Softmax Attention using Structured Matrices

Learnable Low-Rank Polynomial Sketch for Effective Linear Attention

Don't Read Everything: A Curvature-Conditioned Query for Linear Attention

The Entropic Bound for Transformers: Why Static Rank Fails and Attention-Native Rank Recovers