arXiv Machine Learning
Length Generalization for Transformers via Compression
Read the original on arXiv Machine Learning →The Flow has not summarised this story yet — read it at arXiv Machine Learning.
Related stories
arXiv Machine Learning
Jun 2
Length Generalization Bounds for Transformers
arXiv:2603. 02238v2 Announce Type: replace Abstract: Length generalization is a key property of a learning algorithm that enables it to make correct predictions on inputs of any length, given finite training data.
arXiv AI
Aug 14
Algebraic Decomposition Theory for Transformer Length Generalization
arXiv:2608. 13433v1 Announce Type: cross Abstract: Transformer-based language models are known to sometimes generalize to sequences longer than seen during training, but we lack a precise characterization of which tasks admit length generalization.
arXiv AI
Jul 7
On the Ability of Transformers to Verify Plans
arXiv:2603. 19954v2 Announce Type: replace Abstract: Transformers have shown inconsistent success in AI planning tasks, and theoretical understanding of when generalization should be expected has been limited.
arXiv Machine Learning
Jul 14
From Expressivity to Sample Complexity: Narrow Teachers for Transformers via C-RASP
arXiv:2607. 11760v1 Announce Type: new Abstract: A theoretical understanding of Transformers is crucial to better understand the capacities and limitations of large language models (LLMs).
arXiv Machine Learning
Aug 11
Barriers to Universal Reasoning With Transformers (And How to Overcome Them)
arXiv:2604. 25800v2 Announce Type: replace Abstract: Chain-of-Thought (CoT) has been shown to empirically improve Transformers' performance, and theoretically increase their expressivity to Turing completeness.
arXiv Machine Learning
Aug 11
How Many Different Outputs Can a Transformer Generate?
arXiv:2605. 22223v2 Announce Type: replace Abstract: We study how we can leverage only a handful of characteristics of a transformer's architecture to closely predict the number of different sequences it can output, both qualitatively and quantitatively.