arXiv AI

Transformers Can Implement Preconditioned Richardson Iteration for In-Context Gaussian Kernel Regression

arXiv:2605. 08475v3 Announce Type: replace-cross Abstract: In this paper, we study in-context kernel ridge regression (KRR) with Gaussian kernels and show, both theoretically and empirically, that a standard softmax-attention transformer can approximate the KRR predictor during its forward pass.

arXiv Machine Learning
Aug 27

Cubit: Token Mixer with Kernel Ridge Regression

The paper introduces Cubit, a Transformer‑style architecture that replaces the standard attention mechanism with Kernel Ridge Regression (KRR). By interpreting attention as Nadaraya‑Watson regression, Cubit incorporates the closed‑form KRR solution, combining kernel‑based value aggregation with normalization via the inverse kernel matrix. The authors also propose a Limited‑Range Rescale (LRR) to stabilize training and report that Cubit shows improved long‑sequence modeling, with gains increasing as training sequence length grows.

By Chuanyang Zheng, Jiankai Sun, Yihang Gao, Yuehao Wang, Liangchen Tan, Mac Schwager, Anderson Schneider, Yuriy Nevmyvaka, Xiaodong Liu
arXiv Machine Learning
Nov 6

Beyond Quadratic Attention: Separable Gaussian Operators for ODE Vision Transformers

The paper introduces SSoG, a separable sum‑of‑Gaussians spatial operator that replaces dense self‑attention in Vision Transformers, reducing per‑layer cost from ≠N² to subquadratic in token count. SSoG’s fixed Gaussian kernels make it globally Lipschitz continuous, guaranteeing a unique solution for continuous‑depth models, unlike standard dot‑product attention. Experiments on CIFAR‑100 and high‑resolution inputs show that SSoG improves Top‑1 accuracy by up to 8.1 points while using 8.8–16.6% fewer parameters and achieving significantly higher throughput and lower memory usage compared to dense attention.

By Carlos Boned Riera, David Romero Sanchez, Oriol Ramos Terrades