arXiv Machine Learning June 2, 2026 By Andy Yang, Pascal Bergstr\"a{\ss}er, Georg Zetzsche, David Chiang, Anthony W. Lin

Length Generalization Bounds for Transformers

Read the original on arXiv Machine Learning →

arXiv:2603. 02238v2 Announce Type: replace Abstract: Length generalization is a key property of a learning algorithm that enables it to make correct predictions on inputs of any length, given finite training data.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

llms

arXiv Machine Learning

Sep 10

Length Generalization for Transformers via Compression

arXiv:2609.08851v1 Announce Type: new Abstract: Recent advancements in transformer length generalization theory enable us to reliably predict when a transformer can learn to solve a task. In particul...

By Georg Zetzsche, Hongjian Jiang, Andy Yang, Pascal Bergstr\"a{\ss}er, Marco S\"alzer, David Chiang, Anthony W. Lin

llms

arXiv Machine Learning

Jul 14

From Expressivity to Sample Complexity: Narrow Teachers for Transformers via C-RASP

arXiv:2607. 11760v1 Announce Type: new Abstract: A theoretical understanding of Transformers is crucial to better understand the capacities and limitations of large language models (LLMs).

By Michael Rizvi-Martel, Satwik Bhattamishra, Guillaume Rabusseau, Michael Hahn

llms

arXiv AI

Aug 14

Algebraic Decomposition Theory for Transformer Length Generalization

arXiv:2608. 13433v1 Announce Type: cross Abstract: Transformer-based language models are known to sometimes generalize to sequences longer than seen during training, but we lack a precise characterization of which tasks admit length generalization.

By Andy Yang, Blerta Veseli, Corentin Barloy, Micha\"el Cadilhac, Andreas Krebs, Charles Paperman, Howard Straubing, Michael Hahn

llms

arXiv Machine Learning

Sep 10

On Generalisation Error Bounds for Transformers

arXiv:2410. 11500v2 Announce Type: replace-cross Abstract: In this paper, we establish a collection of covering number bounds for linear function classes under various norm constraints on the inputs and matrices.

By Lan V. Truong

llms

arXiv Machine Learning

Sep 1

Universal Transformers for Circuit Computations: Perfect Length Generalization in Tiny Transformers

arXiv:2608.31067v1 Announce Type: new Abstract: Learning generalizable algorithmic computations remains a challenge for neural networks, as reflected in persistent failures on compositional and lengt...

By Takuya Ito, Ruchir Puri, Murray Campbell, Parikshit Ram

llms agents benchmarks

arXiv AI

Jul 7

On the Ability of Transformers to Verify Plans

arXiv:2603. 19954v2 Announce Type: replace Abstract: Transformers have shown inconsistent success in AI planning tasks, and theoretical understanding of when generalization should be expected has been limited.

By Yash Sarrof, Yupei Du, Katharina Stein, Alexander Koller, Sylvie Thi\'ebaux, Michael Hahn

llms

One email a morning, machine-written

Related stories

Length Generalization for Transformers via Compression

From Expressivity to Sample Complexity: Narrow Teachers for Transformers via C-RASP

Algebraic Decomposition Theory for Transformer Length Generalization

On Generalisation Error Bounds for Transformers

Universal Transformers for Circuit Computations: Perfect Length Generalization in Tiny Transformers

On the Ability of Transformers to Verify Plans