arXiv AI
Jul 8
Transformers converge to invariant algorithmic cores
arXiv:2602. 22600v2 Announce Type: replace-cross Abstract: Training selects for behavior, not circuitry: many weight configurations can implement the same function.
Transformers Can Learn Rules They've Never Seen: Proof of Computation Beyond Interpolation
arXiv:2603. 17019v2 Announce Type: replace Abstract: A central question in the debate over large language models is whether transformers can learn rules they have never seen, or whether they can only interpolate: predict new cases from their similarity to training examples.
Related stories
arXiv Machine Learning
Aug 7
Transformers with RL or SFT Provably Learn Sparse Boolean Functions, But Differently
arXiv:2511. 17852v3 Announce Type: replace Abstract: Transformers can acquire Chain-of-Thought (CoT) capabilities to solve reasoning tasks via fine-tuning.
arXiv Machine Learning
Sep 1
Universal Transformers for Circuit Computations: Perfect Length Generalization in Tiny Transformers
arXiv:2608.31067v1 Announce Type: new Abstract: Learning generalizable algorithmic computations remains a challenge for neural networks, as reflected in persistent failures on compositional and lengt...
arXiv Machine Learning
Aug 11
Barriers to Universal Reasoning With Transformers (And How to Overcome Them)
arXiv:2604. 25800v2 Announce Type: replace Abstract: Chain-of-Thought (CoT) has been shown to empirically improve Transformers' performance, and theoretically increase their expressivity to Turing completeness.
arXiv Computation and Language
4d ago
Pretraining Latent Information Feedback Transformers with Teacher Supervision
arXiv:2609.38149v1 Announce Type: new Abstract: Transformer language models (LMs) are feed-forward: deep-layer representations are never fed back to shallower layers, and the only pathway for informa...
arXiv AI
Jul 14
Invariant Learning Dynamics of Transformers in Inductive Reasoning Tasks
arXiv:2607. 11875v1 Announce Type: cross Abstract: We present a theoretical framework to explain the emergence of inductive reasoning abilities in Transformer language models.
arXiv AI
Jun 2
Breaking the Reversal Curse in Autoregressive Language Models via Identity Bridge
arXiv:2602. 02470v2 Announce Type: replace Abstract: Autoregressive large language models (LLMs) have achieved remarkable success in many complex tasks, yet they can still fail in very simple logical reasoning such as the "reversal curse" -- when trained on forward knowledge data of the form "$A \rightarrow B$" (e.
arXiv Machine Learning
Jul 1
Explicit Fuzzy Logic in the Feed-Forward Layer: Self-Forgetting Quantifiers Discover Legible Grammatical-Licensing Detectors
arXiv:2606. 31845v1 Announce Type: cross Abstract: A transformer's feed-forward (FFN) sublayer materializes the distinctions attention gathers, yet gives no account of what it computes.
arXiv Machine Learning
Sep 10
Length Generalization for Transformers via Compression
arXiv:2609.08851v1 Announce Type: new Abstract: Recent advancements in transformer length generalization theory enable us to reliably predict when a transformer can learn to solve a task. In particul...
arXiv AI
Jul 22
The Information Shadow: Measuring Structural Limits on What Language Models Can Learn
arXiv:2607. 18305v1 Announce Type: cross Abstract: Some limits on what language models know are not gaps in data coverage but structural properties of learning from text.
arXiv Machine Learning
Jul 21
Planning with Transformers: Chain of Computation and Structured Context Windows
arXiv:2607. 17710v1 Announce Type: new Abstract: Large Language Models (LLMs) have had a remarkable impact across many areas of machine learning.
arXiv AI
Jul 28
Hierarchical Grading in Large Language Models
arXiv:2607. 22757v1 Announce Type: cross Abstract: We introduce Graded Large Language Models (GLLMs), an algebraic framework that equips the representation space of a transformer with a grading and propagates the induced weighted scalar action through embeddings, self-attention, and the training objective.