arXiv Machine Learning
Length Generalization for Transformers via Compression
Read the original on arXiv Machine Learning →The Flow has not summarised this story yet — read it at arXiv Machine Learning.
Related stories
arXiv Machine Learning
Jun 2
Length Generalization Bounds for Transformers
arXiv:2603. 02238v2 Announce Type: replace Abstract: Length generalization is a key property of a learning algorithm that enables it to make correct predictions on inputs of any length, given finite training data.
arXiv AI
6d ago
Quantitative Generalization Bounds for Multi-Layer Transformers: A Few Long Strings can Beat Many Short Ones
The paper presents quantitative generalization bounds for fixed‑precision transformer models, addressing both in‑distribution statistical generalization and length generalization. It derives guarantees based on model parameters, quantization space, and alphabet size, and shows how training length and sample size trade off, with longer training being exponentially more efficient. Experiments confirm the theoretical predictions.
arXiv AI
Aug 14
Algebraic Decomposition Theory for Transformer Length Generalization
arXiv:2608. 13433v1 Announce Type: cross Abstract: Transformer-based language models are known to sometimes generalize to sequences longer than seen during training, but we lack a precise characterization of which tasks admit length generalization.
arXiv AI
Jul 7
On the Ability of Transformers to Verify Plans
arXiv:2603. 19954v2 Announce Type: replace Abstract: Transformers have shown inconsistent success in AI planning tasks, and theoretical understanding of when generalization should be expected has been limited.
arXiv Machine Learning
Jul 14
From Expressivity to Sample Complexity: Narrow Teachers for Transformers via C-RASP
arXiv:2607. 11760v1 Announce Type: new Abstract: A theoretical understanding of Transformers is crucial to better understand the capacities and limitations of large language models (LLMs).
arXiv Machine Learning
Aug 11
Barriers to Universal Reasoning With Transformers (And How to Overcome Them)
arXiv:2604. 25800v2 Announce Type: replace Abstract: Chain-of-Thought (CoT) has been shown to empirically improve Transformers' performance, and theoretically increase their expressivity to Turing completeness.