arXiv Machine Learning
A Mixture-Gated Theory of Anisotropic Representation Dynamics in Transformers
Read the original on arXiv Machine Learning →The Flow has not summarised this story yet — read it at arXiv Machine Learning.
Related stories
arXiv Machine Learning
Oct 19
Anisotropic Representation Dynamics in Transformers
The paper introduces a Mixture‑Gated Theory of Anisotropic Representation Dynamics that decomposes Transformer self‑attention into value transport, query gating, and key gating. It shows that the geometry of attended representations can cause a representation difference to contract or expand, a phenomenon termed gating‑induced regime switching. The authors derive a spectral criterion to predict switching, quantify its strength, and identify subspaces where switching occurs, validating these predictions experimentally across four Transformer architectures.
arXiv Machine Learning
Sep 1
Tracing distinguishability through transformer processing with stochastic LayerNorm
arXiv:2608.30720v1 Announce Type: new Abstract: Representational similarity is foundational to analyses of deep networks, yet distances between point-valued representations are not intrinsically tied...
arXiv Machine Learning
Jun 19
A Unified Perspective on the Dynamics of Deep Transformers
arXiv:2501. 18322v2 Announce Type: replace Abstract: Transformers, which are state-of-the-art in most machine learning tasks, represent the data as sequences of vectors called tokens.
arXiv AI
Sep 17
The Attention Within: Consensus Dynamics in Selective State Space Models
Selective state space models (SSMs) use a recurrence to mix token information, a process analogous to attention in transformers. By modeling token evolution as an ordinary differential equation and applying input‑to‑state stability, the study proves that SSMs exhibit local exponential stability of consensus equilibria and delineates their domain of attraction for time‑varying weight matrices. Experiments on a pretrained Mamba‑2 model reveal that the output gate controls the degree of consensus, preventing tokens from fully converging.
arXiv Machine Learning
Oct 19
Training-Induced Escape from Token Clustering in a Mean-Field Formulation of Transformers
arXiv:2605.07772v2 Announce Type: replace Abstract: Transformers perform inference by iteratively transforming token representations across layers. This layerwise computation has been studied empiric...
arXiv Machine Learning
Jul 20
Relevant and Irrelevant: A Renormalization Group Analysis of Transformer Attention
arXiv:2607. 15449v1 Announce Type: new Abstract: Using the language of Wilsonian renormalization group theory (RG), we treat the Transformer's attention mechanism as a perturbation of the trained MLP residual-stack fixed point and ask whether it constitutes a relevant, marginal, or irrelevant operator.