← Back to all news
arXiv Machine Learning By Zi Yang, Ying Li, Alejandro Lancho, Dar\'{i}o C. Larese, Pablo M. Olmos, Michael Minyi Zhang

Flow-based Spectral Kernel Learning for Nonstationary Attention

Read the original on arXiv Machine Learning →

The paper introduces Flow-based Spectral Kernel Attention (FSKA), a framework that learns nonstationary attention kernels using spectral density learning and normalizing flows. FSKA replaces traditional query–key projections with a simplified design that retains fixed orthogonal value projections, leading to a more parameter‑efficient architecture. Experiments on classification tasks demonstrate competitive or improved performance while reducing trainable projection parameters by roughly 40%–47% and preserving linear‑attention memory scaling.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

Related stories

arXiv Machine Learning
Aug 27

Cubit: Token Mixer with Kernel Ridge Regression

The paper introduces Cubit, a Transformer‑style architecture that replaces the standard attention mechanism with Kernel Ridge Regression (KRR). By interpreting attention as Nadaraya‑Watson regression, Cubit incorporates the closed‑form KRR solution, combining kernel‑based value aggregation with normalization via the inverse kernel matrix. The authors also propose a Limited‑Range Rescale (LRR) to stabilize training and report that Cubit shows improved long‑sequence modeling, with gains increasing as training sequence length grows.

By Chuanyang Zheng, Jiankai Sun, Yihang Gao, Yuehao Wang, Liangchen Tan, Mac Schwager, Anderson Schneider, Yuriy Nevmyvaka, Xiaodong Liu
llms
More like this →
arXiv Machine Learning
Oct 3

Attention Kernels for Learning Maps Between Heavy-Tailed Measures

The paper introduces attention kernels that replace the exponential function in transformer softmax to better handle operator learning on probability measures with heavy-tailed (polynomial) distributions. Two new benchmarks with closed‑form targets are constructed to evaluate how different kernel growth rates and data preprocessing affect performance. The study finds that slower‑growing kernels prevent ensemble collapse on heavy‑tailed tasks, while softmax with symlog preprocessing only succeeds on a subset of problems, and that all kernels perform similarly on Gaussian data.

By Kailen Hargenrader, Edoardo Calvello, Bohan Chen
llmsbenchmarks
More like this →