arXiv AI
Jun 29
Symmetry-Aware Transformer Training for Automated Planning
arXiv:2508. 07743v2 Announce Type: replace Abstract: While transformers excel in many settings, their application in the field of automated planning is limited.
On the Ability of Transformers to Verify Plans
arXiv:2603. 19954v2 Announce Type: replace Abstract: Transformers have shown inconsistent success in AI planning tasks, and theoretical understanding of when generalization should be expected has been limited.
Related stories
arXiv Machine Learning
Jul 14
From Expressivity to Sample Complexity: Narrow Teachers for Transformers via C-RASP
arXiv:2607. 11760v1 Announce Type: new Abstract: A theoretical understanding of Transformers is crucial to better understand the capacities and limitations of large language models (LLMs).
arXiv Machine Learning
Jun 2
Length Generalization Bounds for Transformers
arXiv:2603. 02238v2 Announce Type: replace Abstract: Length generalization is a key property of a learning algorithm that enables it to make correct predictions on inputs of any length, given finite training data.
arXiv AI
Jun 8
Discovering Interpretable Algorithms by Decompiling Transformers to RASP
arXiv:2602. 08857v2 Announce Type: replace-cross Abstract: Recent work has shown that the computations of Transformers can be simulated in the RASP family of programming languages.
arXiv Machine Learning
Jul 21
Planning with Transformers: Chain of Computation and Structured Context Windows
arXiv:2607. 17710v1 Announce Type: new Abstract: Large Language Models (LLMs) have had a remarkable impact across many areas of machine learning.
arXiv Machine Learning
Aug 11
Barriers to Universal Reasoning With Transformers (And How to Overcome Them)
arXiv:2604. 25800v2 Announce Type: replace Abstract: Chain-of-Thought (CoT) has been shown to empirically improve Transformers' performance, and theoretically increase their expressivity to Turing completeness.
arXiv Machine Learning
Aug 11
How Many Different Outputs Can a Transformer Generate?
arXiv:2605. 22223v2 Announce Type: replace Abstract: We study how we can leverage only a handful of characteristics of a transformer's architecture to closely predict the number of different sequences it can output, both qualitatively and quantitatively.
arXiv AI
Jul 28
How Transformers Learn to Plan via Multi-Token Prediction
arXiv:2604. 11912v2 Announce Type: replace-cross Abstract: While next-token prediction (NTP) has been the standard objective for training language models, it often struggles to capture global structure in reasoning tasks.
arXiv Machine Learning
Jul 21
Can Transformers Really Do It All? On the Compatibility of Inductive Biases Across Tasks
arXiv:2607. 17624v1 Announce Type: new Abstract: Transformers are remarkably versatile and their design is largely consistent across a variety of applications.
arXiv Machine Learning
Jul 30
Transformers Can Learn Rules They've Never Seen: Proof of Computation Beyond Interpolation
arXiv:2603. 17019v2 Announce Type: replace Abstract: A central question in the debate over large language models is whether transformers can learn rules they have never seen, or whether they can only interpolate: predict new cases from their similarity to training examples.
arXiv AI
5d ago
On the Expressive Power of Transformers
arXiv:2608. 12671v1 Announce Type: new Abstract: Multi-layer transformers form the critical component of essentially all large language models (LLMs) in use today.
arXiv Machine Learning
Jun 9
Understanding the Parameter Space Geometry of Transformers Encoding Boolean Functions
arXiv:2606. 08768v1 Announce Type: new Abstract: Transformers consistently fail to learn certain simple functions that are provably expressible with specific parameter settings.