arXiv Machine Learning July 28, 2026 By Byeong Hoon Yoon

The Entropic Bound for Transformers: Why Static Rank Fails and Attention-Native Rank Recovers

Read the original on arXiv Machine Learning →

arXiv:2607. 23050v1 Announce Type: new Abstract: Neural scaling laws describe how loss decreases as models, data, and compute grow, but they do not answer a prior question: for a fixed task, what is the minimum model capacity required to solve it?

Summary generated by The Flow from the publisher's feed. The full article lives at arXiv Machine Learning.

arXiv Machine Learning

Aug 11

Training-Free Universal Approximation by Prompting Random Transformers

arXiv:2608. 09558v1 Announce Type: new Abstract: How expressive is prompting a transformer?

By Alexander Hsu, Rongjie Lai

llms

arXiv Machine Learning

Jul 22

Relative Positions Generalize, Absolute Positions Memorize: An Implicit-Bias Account of Length Generalization in Attention

arXiv:2607. 18759v1 Announce Type: new Abstract: Transformers with relative positional encodings often extrapolate to sequences longer than those seen during training, whereas transformers with learned absolute encodings typically do not.

By Subham Singh, Ashutosh Mishra, Subha Raut

llms safety

arXiv Machine Learning

Jun 17

Softmax as Linear Attention in the Large-Prompt Regime: a Measure-based Perspective

arXiv:2512. 11784v2 Announce Type: replace Abstract: Softmax attention is a central component of transformer architectures, yet its nonlinear structure poses significant challenges for theoretical analysis.

By Etienne Boursier, Claire Boyer

llms

arXiv AI

1d ago

Exact Attention Sensitivity and the Geometry of Transformer Stability

arXiv:2602. 18849v2 Announce Type: replace-cross Abstract: We develop a sensitivity analysis for transformer attention in a geometry aligned with tokenwise computation.

By Seyed Morteza Emadi

llms

arXiv AI

Jul 23

ELSAA: Efficient Low-Rank and Sparse Attention Approximation for Training Transformers

arXiv:2607. 20214v1 Announce Type: cross Abstract: The quadratic $N\times N$ attention score matrix remains a central obstacle to extending Transformers to longer input lengths.

By Mahdi Heidari, Mohammad Mahdi Rahimi, Jaekyun Moon

llms efficiency

arXiv Machine Learning

Jun 2

Don't Read Everything: A Curvature-Conditioned Query for Linear Attention

arXiv:2606. 01294v1 Announce Type: cross Abstract: Linear attention reduces the quadratic cost of softmax attention by maintaining a recurrent fast-weight state, but it consistently lags on in-context retrieval and long-context tasks.

By Dong Le, Thong Nguyen, Cong-Duy Nguyen, Anh Tuan Luu

Related stories

Training-Free Universal Approximation by Prompting Random Transformers

Relative Positions Generalize, Absolute Positions Memorize: An Implicit-Bias Account of Length Generalization in Attention

Softmax as Linear Attention in the Large-Prompt Regime: a Measure-based Perspective

Exact Attention Sensitivity and the Geometry of Transformer Stability

ELSAA: Efficient Low-Rank and Sparse Attention Approximation for Training Transformers

Don't Read Everything: A Curvature-Conditioned Query for Linear Attention