arXiv AI
The Long Delay to Arithmetic Generalization
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv AI
Jun 18
The Long Delay to Arithmetic Generalization: When Learned Representations Outrun Behavior
arXiv:2604. 13082v2 Announce Type: replace-cross Abstract: Grokking in transformers trained on algorithmic tasks is characterized by a long delay between training-set fit and abrupt generalization, but the source of that delay remains poorly understood.
arXiv AI
2d ago
Quantitative Generalization Bounds for Multi-Layer Transformers: A Few Long Strings can Beat Many Short Ones
The paper presents quantitative generalization bounds for fixed‑precision transformer models, addressing both in‑distribution statistical generalization and length generalization. It derives guarantees based on model parameters, quantization space, and alphabet size, and shows how training length and sample size trade off, with longer training being exponentially more efficient. Experiments confirm the theoretical predictions.
arXiv AI
3d ago
Testing Transformer Learnability on the Arithmetic Sequence of Rooted Trees
arXiv:2512. 01870v3 Announce Type: replace Abstract: We study whether a transformer network can learn the deterministic sequence of trees generated by the iterated prime factorization of the natural numbers.
arXiv Machine Learning
Sep 10
Length Generalization for Transformers via Compression
arXiv:2609.08851v1 Announce Type: new Abstract: Recent advancements in transformer length generalization theory enable us to reliably predict when a transformer can learn to solve a task. In particul...
arXiv Machine Learning
Jul 30
Transformers Can Learn Rules They've Never Seen: Proof of Computation Beyond Interpolation
arXiv:2603. 17019v2 Announce Type: replace Abstract: A central question in the debate over large language models is whether transformers can learn rules they have never seen, or whether they can only interpolate: predict new cases from their similarity to training examples.
arXiv Machine Learning
Sep 1
Universal Transformers for Circuit Computations: Perfect Length Generalization in Tiny Transformers
arXiv:2608.31067v1 Announce Type: new Abstract: Learning generalizable algorithmic computations remains a challenge for neural networks, as reflected in persistent failures on compositional and lengt...