arXiv Machine Learning August 4, 2026 By Rahel Rickenbach, Jelena Trisovic, Alexandre Didier, Jerome Sieber, Melanie N. Zeilinger

Eigenvalues as a Metric for Memory Dynamics in Sequence Models

Read the original on arXiv Machine Learning →

arXiv:2510. 09379v2 Announce Type: replace Abstract: While softmax attention drives state-of-the-art performance in sequence modeling, its quadratic complexity motivates linear alternatives such as state space models (SSMs).

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

benchmarks

arXiv AI

Jun 10

Dynamic Linear Attention

arXiv:2606. 10650v1 Announce Type: cross Abstract: The scalability of Large Language Models (LLMs) to long contexts is fundamentally constrained by the quadratic complexity of standard attention, motivating the adoption of linear attention mechanisms with sub-quadratic cost.

By Xin Wang, Hui Shen, Boyuan Zheng, Xueshen Liu, Minkyoung Cho, Zhongwei Wan, Zesen Zhao, Zhuoqing Mao, Shen Yan, Mi Zhang

llms benchmarks

arXiv Computation and Language

3d ago

The Evolution of Attention in Large Language Models: Mechanisms, Trade-offs, and Emerging Trends

arXiv:2609.39661v1 Announce Type: new Abstract: Self-attention gives LLMs fine-grained, query-dependent access to context, but dense token interactions incur quadratic prefill cost and a key--value c...

By Zhentao Tan, Jingyi Shen, Yanbo Li, Yao Liu, Yue Wu, Jieping Ye

llms

Hugging Face Trending Papers

Jun 9

Dynamic Linear Attention

The scalability of Large Language Models (LLMs) to long contexts is fundamentally constrained by the quadratic complexity of standard attention, motivating the adoption of linear attention mechanisms with sub-quadratic cost. To improve representation capacity under long contexts, recent approaches organize memory in a multi-state manner.

llms benchmarks

arXiv Machine Learning

Jun 11

On Subquadratic Architectures: From Applications to Principles

arXiv:2606. 12364v1 Announce Type: new Abstract: Transformers dominate modern sequence modeling, but their quadratic attention incurs substantial computational cost.

By Anamaria-Roberta Hartl, Levente Z\'olyomi, David Stap, Pieter-Jan Hoedt, Niklas Schmidinger, Lukas Hauzenberger, Sebastian B\"ock, G\"unter Klambauer, Sepp Hochreiter

llms efficiency

arXiv AI

Jun 2

Learning to Remember, Learn, and Forget in Attention-Based Models

arXiv:2602. 09075v3 Announce Type: replace-cross Abstract: In-Context Learning (ICL) in transformers acts as an online associative memory and is believed to underpin their high performance on complex sequence processing tasks.

By Djohan Bonnet, Jamie Lohoff, Jan Finkbeiner, Elidona Skhikerujah, Emre Neftci

llms benchmarks

arXiv Machine Learning

Aug 11

Advancing Intelligent Sequence Modeling: Evolution, Trade-offs, and Applications of State-Space Architectures from S4 to Mamba

arXiv:2503. 18970v4 Announce Type: replace Abstract: Structured State Space Models (SSMs) have become a prominent class of sequence models, developed against two long-standing difficulties: the sequential computation and gradient propagation limits of Recurrent Neural Networks (RNNs), and the quadratic time and memory cost of self-attention in Transformers.

By Shriyank Somvanshi, Md Monzurul Islam, Mahmuda Sultana Mimi, Sazzad Bin Bashar Polock, Gaurab Chhetri, Anandi Dutta, Amir Rafe, Subasish Das

llms

One email a morning, machine-written

Related stories

Dynamic Linear Attention

The Evolution of Attention in Large Language Models: Mechanisms, Trade-offs, and Emerging Trends

Dynamic Linear Attention

On Subquadratic Architectures: From Applications to Principles

Learning to Remember, Learn, and Forget in Attention-Based Models

Advancing Intelligent Sequence Modeling: Evolution, Trade-offs, and Applications of State-Space Architectures from S4 to Mamba