arXiv:2605. 10931v2 Announce Type: replace-cross Abstract: Transformers with self-attention modules as their core components have become an integral architecture in modern large language and foundation models.
By Albert Alcalde, Leon Bungert, Konstantin Riedl, Tim Roith
arXiv:2607. 15449v1 Announce Type: new Abstract: Using the language of Wilsonian renormalization group theory (RG), we treat the Transformer's attention mechanism as a perturbation of the trained MLP residual-stack fixed point and ask whether it constitutes a relevant, marginal, or irrelevant operator.
By Parviz Haggi-Mani, Irina Rish
arXiv:2610.17347v1 Announce Type: cross
Abstract: We revisit the statistical mechanics of Transformer attention via modern Hopfield models. We present a free-energy formalism to characterize paramete...
By Jerry Yao-Chieh Hu, Yu-An Chen, Maojiang Su, Han Liu
arXiv:2501. 18322v2 Announce Type: replace Abstract: Transformers, which are state-of-the-art in most machine learning tasks, represent the data as sequences of vectors called tokens.
By Val\'erie Castin, Pierre Ablin, Jos\'e Antonio Carrillo, Gabriel Peyr\'e
The paper introduces a Mixture‑Gated Theory of Anisotropic Representation Dynamics that decomposes Transformer self‑attention into value transport, query gating, and key gating. It shows that the geometry of attended representations can cause a representation difference to contract or expand, a phenomenon termed gating‑induced regime switching. The authors derive a spectral criterion to predict switching, quantify its strength, and identify subspaces where switching occurs, validating these predictions experimentally across four Transformer architectures.
By Nguyen Do, My T. Thai
arXiv:2610.23242v2 Announce Type: replace
Abstract: Small differences between internal representations can shrink or grow as they pass through a Transformer layer, affecting how information is preser...
By Nguyen Do, My T. Thai