← Back to all news
arXiv Machine Learning By Chuanyang Zheng, Jiankai Sun, Yihang Gao, Yuehao Wang, Liangchen Tan, Mac Schwager, Anderson Schneider, Yuriy Nevmyvaka, Xiaodong Liu

Cubit: Token Mixer with Kernel Ridge Regression

Read the original on arXiv Machine Learning →

The paper introduces Cubit, a Transformer‑style architecture that replaces the standard attention mechanism with Kernel Ridge Regression (KRR). By interpreting attention as Nadaraya‑Watson regression, Cubit incorporates the closed‑form KRR solution, combining kernel‑based value aggregation with normalization via the inverse kernel matrix. The authors also propose a Limited‑Range Rescale (LRR) to stabilize training and report that Cubit shows improved long‑sequence modeling, with gains increasing as training sequence length grows.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

Related stories

Hugging Face Trending Papers
Oct 9

Data-Adaptive Mahalanobis Metric Learning for Cross-Head Attention in Transformers

The paper introduces Data-Adaptive Mahalanobis Cross-Head Attention (DAMCHA), a new attention mechanism that extends Multi-Head Attention by learning a full cross-head metric matrix, thereby capturing inter-head interactions beyond the block-diagonal structure of standard MHA. DAMCHA parameterizes this metric as an input-dependent function, allowing the attention geometry to adapt dynamically to the data’s intrinsic manifold structure. The authors also propose regularization techniques, such as stack-wise parameter sharing, to maintain computational efficiency and stable optimization, and demonstrate that DAMCHA-equipped Transformers outperform existing baselines on multiple benchmark tasks.

llmsbenchmarks
More like this →
arXiv Machine Learning
Oct 12

Data-Adaptive Mahalanobis Metric Learning for Cross-Head Attention in Transformers

The paper introduces Data-Adaptive Mahalanobis Cross-Head Attention (DAMCHA), a new attention mechanism that extends Multi-Head Attention by learning a full cross-head metric matrix and adapting it to input data. DAMCHA captures inter-head interactions beyond the block-diagonal structure of standard MHA and uses input-dependent parameterization for dynamic attention geometry. The authors provide regularization techniques, such as stack-wise parameter sharing, to maintain efficiency and stability, and report that DAMCHA-based Transformers outperform baselines on several benchmarks.

By Zhenke Duan, Xin Li, Jiqun Pan, Dong Xiaofei, Hanwen Ning, Xinyuan Song
llmsbenchmarks
More like this →