arXiv Machine Learning July 22, 2026 By Subham Singh, Ashutosh Mishra, Subha Raut

Relative Positions Generalize, Absolute Positions Memorize: An Implicit-Bias Account of Length Generalization in Attention

Read the original on arXiv Machine Learning →

arXiv:2607. 18759v1 Announce Type: new Abstract: Transformers with relative positional encodings often extrapolate to sequences longer than those seen during training, whereas transformers with learned absolute encodings typically do not.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

arXiv Machine Learning

Jul 28

The Entropic Bound for Transformers: Why Static Rank Fails and Attention-Native Rank Recovers

arXiv:2607. 23050v1 Announce Type: new Abstract: Neural scaling laws describe how loss decreases as models, data, and compute grow, but they do not answer a prior question: for a fixed task, what is the minimum model capacity required to solve it?

By Byeong Hoon Yoon

llms safety

arXiv Machine Learning

Jun 2

Don't Read Everything: A Curvature-Conditioned Query for Linear Attention

arXiv:2606. 01294v1 Announce Type: cross Abstract: Linear attention reduces the quadratic cost of softmax attention by maintaining a recurrent fast-weight state, but it consistently lags on in-context retrieval and long-context tasks.

By Dong Le, Thong Nguyen, Cong-Duy Nguyen, Anh Tuan Luu

arXiv AI

Jun 18

Dual Dimensionality for Local and Global Attention

arXiv:2606. 18587v1 Announce Type: cross Abstract: Decoder-only Transformers compute attention over the KV cache of preceding tokens.

By Zhiyuan Wang, Xuan Luo, Sirui Zeng, Xifeng Yan

llms fine-tuning efficiency

arXiv Machine Learning

Jun 17

Softmax as Linear Attention in the Large-Prompt Regime: a Measure-based Perspective

arXiv:2512. 11784v2 Announce Type: replace Abstract: Softmax attention is a central component of transformer architectures, yet its nonlinear structure poses significant challenges for theoretical analysis.

By Etienne Boursier, Claire Boyer

llms

arXiv Machine Learning

Aug 11

Training-Free Universal Approximation by Prompting Random Transformers

arXiv:2608. 09558v1 Announce Type: new Abstract: How expressive is prompting a transformer?

By Alexander Hsu, Rongjie Lai

llms

arXiv AI

Jul 22

Soft-TransFormers for Continual Learning

arXiv:2411. 16073v4 Announce Type: replace-cross Abstract: Inspired by the Well-initialized Lottery Ticket Hypothesis (WLTH), we introduce Soft-TransFormers (Soft-TF), a continual learning framework that adapts a frozen pre-trained Transformer through task-specific soft subnetworks: real-valued multiplicative masks over the query, key, value, and output projections of selected self-attention layers.

By Haeyong Kang, Chang D. Yoo

llms fine-tuning benchmarks

One email a morning, machine-written

Related stories

The Entropic Bound for Transformers: Why Static Rank Fails and Attention-Native Rank Recovers

Don't Read Everything: A Curvature-Conditioned Query for Linear Attention

Dual Dimensionality for Local and Global Attention

Softmax as Linear Attention in the Large-Prompt Regime: a Measure-based Perspective

Training-Free Universal Approximation by Prompting Random Transformers

Soft-TransFormers for Continual Learning