arXiv Machine Learning June 5, 2026 By Weinuo Ou

Exact Linear Attention

Read the original on arXiv Machine Learning →

arXiv:2605. 18848v3 Announce Type: replace Abstract: This paper introduces Exact Linear Attention (ELA), a mechanism that achieves linear computational complexity for Transformer attention by exploiting the exact decomposition property of kernel functions, thereby eliminating approximation error.

Summary generated by The Flow from the publisher's feed. The full article lives at arXiv Machine Learning.

arXiv AI

Aug 10

Learning to Predict Middle-Layer Attention in MLLMs for Visual Token Prunin

arXiv:2608. 06411v1 Announce Type: new Abstract: Multimodal large language models (MLLMs) achieve strong performance across diverse vision-language tasks, but their efficiency is limited by the cost of processing numerous visual tokens.

By Yuyao Sun, Tao Deng, Shuang Li, Deqing Wang, Hao Geng, Minjun Yu

llms efficiency multimodal benchmarks

arXiv AI

Jun 18

Dual Dimensionality for Local and Global Attention

arXiv:2606. 18587v1 Announce Type: cross Abstract: Decoder-only Transformers compute attention over the KV cache of preceding tokens.

By Zhiyuan Wang, Xuan Luo, Sirui Zeng, Xifeng Yan

llms fine-tuning efficiency

arXiv AI

Jul 23

LISA: Linear-Indexed Sparse Attention for Efficient Long-Context Reasoning

arXiv:2607. 19358v1 Announce Type: new Abstract: Recent advances in long chain-of-thought reasoning models such as DeepSeek-R1 have led to increasingly longer inference context lengths under the test-time scaling paradigm.

By Yu Zhao, Zekun Zhang, Fan Jiang, Bo Zeng, Linlong Xu, Shimin Shan, Yu Liu, Longyue Wang, Weihua Luo

efficiency benchmarks

arXiv AI

Jul 23

ELSAA: Efficient Low-Rank and Sparse Attention Approximation for Training Transformers

arXiv:2607. 20214v1 Announce Type: cross Abstract: The quadratic $N\times N$ attention score matrix remains a central obstacle to extending Transformers to longer input lengths.

By Mahdi Heidari, Mohammad Mahdi Rahimi, Jaekyun Moon

llms efficiency

arXiv AI

5d ago

Reduced Matrix Multiplication: Input-Adaptive Matrix-Product Reduction for LLM Inference

arXiv:2608. 13426v1 Announce Type: cross Abstract: Transformer-based language models achieve strong performance but incur substantial inference cost due to repeated high-dimensional matrix multiplications.

By Zixuan Lan, Yanhong Li, Jiawei Zhou

llms multimodal benchmarks

arXiv Machine Learning

Jul 2

Ghost in the Kernel: In-Context Learning with Efficient Transformers via Domain Generalization

arXiv:2607. 00479v1 Announce Type: new Abstract: Transformer-based large models have demonstrated remarkable generalization abilities across different tasks by leveraging a context-aware attention module for in-context learning.

By Peilin Liu, Ding-Xuan Zhou

llms

Related stories

Learning to Predict Middle-Layer Attention in MLLMs for Visual Token Prunin

Dual Dimensionality for Local and Global Attention

LISA: Linear-Indexed Sparse Attention for Efficient Long-Context Reasoning

ELSAA: Efficient Low-Rank and Sparse Attention Approximation for Training Transformers

Reduced Matrix Multiplication: Input-Adaptive Matrix-Product Reduction for LLM Inference

Ghost in the Kernel: In-Context Learning with Efficient Transformers via Domain Generalization