arXiv Machine Learning August 27, 2026 By Chuanyang Zheng, Jiankai Sun, Yihang Gao, Yuehao Wang, Liangchen Tan, Mac Schwager, Anderson Schneider, Yuriy Nevmyvaka, Xiaodong Liu

Cubit: Token Mixer with Kernel Ridge Regression

Read the original on arXiv Machine Learning →

The paper introduces Cubit, a Transformer‑style architecture that replaces the standard attention mechanism with Kernel Ridge Regression (KRR). By interpreting attention as Nadaraya‑Watson regression, Cubit incorporates the closed‑form KRR solution, combining kernel‑based value aggregation with normalization via the inverse kernel matrix. The authors also propose a Limited‑Range Rescale (LRR) to stabilize training and report that Cubit shows improved long‑sequence modeling, with gains increasing as training sequence length grows.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

llms

arXiv Machine Learning

Jun 30

Generalization Analysis of Transformers in Distribution Regression

arXiv:2606. 29256v1 Announce Type: cross Abstract: In recent years, models based on the Transformer architecture have seen widespread applications and have become one of the core tools in the field of deep learning.

By Peilin Liu, Ding-Xuan Zhou

llms nlp fine-tuning

arXiv Machine Learning

Jun 5

Exact Linear Attention

arXiv:2605. 18848v3 Announce Type: replace Abstract: This paper introduces Exact Linear Attention (ELA), a mechanism that achieves linear computational complexity for Transformer attention by exploiting the exact decomposition property of kernel functions, thereby eliminating approximation error.

By Weinuo Ou

llms computer-vision reinforcement-learning efficiency safety

arXiv AI

Jun 29

Flexformer: Flexible Linear Transformer with Learnable Attention Kernel

arXiv:2606. 27748v1 Announce Type: cross Abstract: Transformer models rely on attention mechanism to capture long-range dependencies but suffer from quadratic complexity, limiting their scalability to long sequences.

By Haoran Zhang, Feng Zhou

llms

arXiv Machine Learning

Jul 2

Ghost in the Kernel: In-Context Learning with Efficient Transformers via Domain Generalization

arXiv:2607. 00479v1 Announce Type: new Abstract: Transformer-based large models have demonstrated remarkable generalization abilities across different tasks by leveraging a context-aware attention module for in-context learning.

By Peilin Liu, Ding-Xuan Zhou

llms

arXiv Machine Learning

Jul 22

Relative Positions Generalize, Absolute Positions Memorize: An Implicit-Bias Account of Length Generalization in Attention

arXiv:2607. 18759v1 Announce Type: new Abstract: Transformers with relative positional encodings often extrapolate to sequences longer than those seen during training, whereas transformers with learned absolute encodings typically do not.

By Subham Singh, Ashutosh Mishra, Subha Raut

llms safety

arXiv AI

Jul 23

ELSAA: Efficient Low-Rank and Sparse Attention Approximation for Training Transformers

arXiv:2607. 20214v1 Announce Type: cross Abstract: The quadratic $N\times N$ attention score matrix remains a central obstacle to extending Transformers to longer input lengths.

By Mahdi Heidari, Mohammad Mahdi Rahimi, Jaekyun Moon

llms efficiency

One email a morning, machine-written

Related stories

Generalization Analysis of Transformers in Distribution Regression

Exact Linear Attention

Flexformer: Flexible Linear Transformer with Learnable Attention Kernel

Ghost in the Kernel: In-Context Learning with Efficient Transformers via Domain Generalization

Relative Positions Generalize, Absolute Positions Memorize: An Implicit-Bias Account of Length Generalization in Attention

ELSAA: Efficient Low-Rank and Sparse Attention Approximation for Training Transformers