arXiv AI
Aug 11
The Transformer Revolution, Part 1: Dynamic Processing through Output-Weight Interconnections
arXiv:2608. 03921v2 Announce Type: replace Abstract: This paper offers a new interpretation of the Transformer during inference.
Hierarchical Latent Structures in Data Generation Process Unify Mechanistic Phenomena across Scale
arXiv:2603. 06592v2 Announce Type: replace-cross Abstract: Contemporary studies in mechanistic interpretability have uncovered many puzzling phenomena in the neural information processing of Transformer-based language models, such as induction heads, function vectors, and the Hydra effect.
Related stories
arXiv AI
Jun 8
The Latent Space: Foundation, Evolution, Mechanism, Ability, and Outlook
arXiv:2604. 02029v2 Announce Type: replace Abstract: Latent space is rapidly emerging as a native substrate for language-based models.
arXiv AI
Aug 5
The Transformer Revolution, Part 1: Dynamic Processing through Output- Weight Interconnections
arXiv:2608. 03921v1 Announce Type: new Abstract: This paper offers a new interpretation of the Transformer during inference.
arXiv Machine Learning
Jun 17
An expressivity analysis of hierarchical modelling in deep transformers via bounded-depth grammars
arXiv:2606. 17522v1 Announce Type: cross Abstract: Deep neural networks are widely believed to derive their expressive power from their ability to form \textbf{hierarchical representations}, capturing progressively more abstract and compositional features across layers.
arXiv AI
Jul 7
Language Models Represent and Transform Concepts with Shared Geometry
arXiv:2607. 04525v1 Announce Type: cross Abstract: How concepts are represented in neural networks is a fundamental question in machine learning.
Hugging Face Trending Papers
Jul 5
Language Models Represent and Transform Concepts with Shared Geometry
How concepts are represented in neural networks is a fundamental question in machine learning. The dominant view treats concept representations as stationary geometric objects.
arXiv Machine Learning
Aug 7
Cross-Architecture Steering Transfer in Language Models: A Systematic Empirical Study
arXiv:2608. 05164v1 Announce Type: cross Abstract: Independently trained large language models may develop shared internal representations of semantic concepts despite architectural differences -- but whether this geometric similarity has functional consequences for cross-model behavioural control remains untested.
arXiv AI
Jul 8
Transformers converge to invariant algorithmic cores
arXiv:2602. 22600v2 Announce Type: replace-cross Abstract: Training selects for behavior, not circuitry: many weight configurations can implement the same function.
arXiv Machine Learning
Jun 9
Trajectory Geometry of Transformer Representations Across Layers
arXiv:2606. 09287v1 Announce Type: new Abstract: Understanding how transformer representations evolve across layers, not merely what they encode, remains an open problem in mechanistic interpretability.
Hugging Face Trending Papers
Jul 13
Invariant Learning Dynamics of Transformers in Inductive Reasoning Tasks
We present a theoretical framework to explain the emergence of inductive reasoning abilities in Transformer language models. While previous works on Transformer learning dynamics have so far been mostly tied to specific tasks, we study a generalized class of inductive tasks that unifies several synthetic tasks known in the literature, including in-context n-grams and multi-hop reasoning.
arXiv AI
Aug 11
Shattered Compositionality: Counterintuitive Learning Dynamics of Transformers for Arithmetic
arXiv:2601. 22510v2 Announce Type: replace-cross Abstract: Large language models (LLMs) often achieve strong benchmark accuracy yet remain brittle under small distribution shifts.
arXiv Machine Learning
Jun 25
Neural Scaling Universality: If Exponents Are Fixed, Time to Understand Coefficients
arXiv:2606. 25008v1 Announce Type: new Abstract: Neural scaling laws describe how pre-training loss decays as power laws with training time, model size, and compute.