arXiv Machine Learning
Training-Induced Escape from Token Clustering in a Mean-Field Formulation of Transformers
Read the original on arXiv Machine Learning →The Flow has not summarised this story yet — read it at arXiv Machine Learning.
Related stories
arXiv Machine Learning
Jun 19
A Unified Perspective on the Dynamics of Deep Transformers
arXiv:2501. 18322v2 Announce Type: replace Abstract: Transformers, which are state-of-the-art in most machine learning tasks, represent the data as sequences of vectors called tokens.
arXiv Machine Learning
Jun 30
A Mechanistic Study of Transformers Training Dynamics
arXiv:2410. 24050v3 Announce Type: replace Abstract: Large-scale pretraining of transformers has been central to the success of foundation models.
arXiv Machine Learning
Oct 8
Leaner Transformers Can Easily Learn to Cluster
arXiv:2610. 09760v1 Announce Type: new Abstract: Transformers have in-context learning capabilities, where some known learning algorithms can be executed in the forward pass through the model.
arXiv Machine Learning
5d ago
Anisotropic Representation Dynamics in Transformers
The paper introduces a Mixture‑Gated Theory of Anisotropic Representation Dynamics that decomposes Transformer self‑attention into value transport, query gating, and key gating. It shows that the geometry of attended representations can cause a representation difference to contract or expand, a phenomenon termed gating‑induced regime switching. The authors derive a spectral criterion to predict switching, quantify its strength, and identify subspaces where switching occurs, validating these predictions experimentally across four Transformer architectures.
arXiv AI
Sep 10
Transformers as In-Context Samplers: From Closed-Form Diffusion to Estimation-Free Sampling
arXiv:2609.08981v1 Announce Type: cross Abstract: A growing body of work establishes that large language models are not mere statistical memorizers, but are capable of in-context learning: performing...
arXiv Machine Learning
Jul 2
Ghost in the Kernel: In-Context Learning with Efficient Transformers via Domain Generalization
arXiv:2607. 00479v1 Announce Type: new Abstract: Transformer-based large models have demonstrated remarkable generalization abilities across different tasks by leveraging a context-aware attention module for in-context learning.