arXiv Machine Learning June 5, 2026 By Weinuo Ou

Exact Linear Attention

Read the original on arXiv Machine Learning →

arXiv:2605. 18848v3 Announce Type: replace Abstract: This paper introduces Exact Linear Attention (ELA), a mechanism that achieves linear computational complexity for Transformer attention by exploiting the exact decomposition property of kernel functions, thereby eliminating approximation error.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

arXiv Computer Vision

2d ago

Do Vision Transformers Need All-to-All Attention? Global Communication Through Elastic Learned Cores

arXiv:2605.12491v2 Announce Type: replace Abstract: Vision Transformers (ViTs) learn rich visual-semantic representations through all-to-all self-attention among patch tokens. However, this design im...

By Alan Z. Song, Yinjie Chen, Mu Nan, Deva Ramanan, Michael J. Tarr, Andrew F. Luo

llms computer-vision benchmarks

arXiv Machine Learning

Aug 27

Cubit: Token Mixer with Kernel Ridge Regression

The paper introduces Cubit, a Transformer‑style architecture that replaces the standard attention mechanism with Kernel Ridge Regression (KRR). By interpreting attention as Nadaraya‑Watson regression, Cubit incorporates the closed‑form KRR solution, combining kernel‑based value aggregation with normalization via the inverse kernel matrix. The authors also propose a Limited‑Range Rescale (LRR) to stabilize training and report that Cubit shows improved long‑sequence modeling, with gains increasing as training sequence length grows.

By Chuanyang Zheng, Jiankai Sun, Yihang Gao, Yuehao Wang, Liangchen Tan, Mac Schwager, Anderson Schneider, Yuriy Nevmyvaka, Xiaodong Liu

llms

arXiv Computer Vision

Sep 1

Fast and Accurate Image Restoration and Generation with Rank Enhanced Linear Attention

arXiv:2505.16157v3 Announce Type: replace Abstract: Transformer-based models have made remarkable progress in image restoration (IR) tasks. However, the quadratic complexity of self-attention in Tran...

By Yuang Ai

llms diffusion computer-vision benchmarks

arXiv Computer Vision

Sep 10

Learning to Predict Middle-Layer Attention in MLLMs for Visual Token Pruning

arXiv:2608.06411v2 Announce Type: replace-cross Abstract: Multimodal large language models (MLLMs) achieve strong performance across diverse vision-language tasks, but their efficiency is limited by...

By Yuyao Sun, Tao Deng, Shuang Li, Deqing Wang, Hao Geng, Minjun Yu

llms efficiency multimodal benchmarks

arXiv AI

Aug 10

Learning to Predict Middle-Layer Attention in MLLMs for Visual Token Prunin

arXiv:2608. 06411v1 Announce Type: new Abstract: Multimodal large language models (MLLMs) achieve strong performance across diverse vision-language tasks, but their efficiency is limited by the cost of processing numerous visual tokens.

By Yuyao Sun, Tao Deng, Shuang Li, Deqing Wang, Hao Geng, Minjun Yu

llms efficiency multimodal benchmarks

arXiv AI

Jun 18

Dual Dimensionality for Local and Global Attention

arXiv:2606. 18587v1 Announce Type: cross Abstract: Decoder-only Transformers compute attention over the KV cache of preceding tokens.

By Zhiyuan Wang, Xuan Luo, Sirui Zeng, Xifeng Yan

llms fine-tuning efficiency

One email a morning, machine-written

Related stories

Do Vision Transformers Need All-to-All Attention? Global Communication Through Elastic Learned Cores

Cubit: Token Mixer with Kernel Ridge Regression

Fast and Accurate Image Restoration and Generation with Rank Enhanced Linear Attention

Learning to Predict Middle-Layer Attention in MLLMs for Visual Token Pruning

Learning to Predict Middle-Layer Attention in MLLMs for Visual Token Prunin

Dual Dimensionality for Local and Global Attention