arXiv AI
On the Statistical Mechanics of Attention
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv Machine Learning
Jun 19
A Unified Perspective on the Dynamics of Deep Transformers
arXiv:2501. 18322v2 Announce Type: replace Abstract: Transformers, which are state-of-the-art in most machine learning tasks, represent the data as sequences of vectors called tokens.
arXiv Machine Learning
Jun 11
Phase Transitions in Attention: A Bayesian Theory of Copy Head Emergence
arXiv:2606. 12058v1 Announce Type: cross Abstract: Attention is the key mechanism underlying in-context learning in transformers, and attention patterns have been observed empirically to emerge abruptly during training.
arXiv AI
Sep 2
Energy-Based Transformers as Predictors of Reading Difficulty
The paper investigates energy-based transformers as predictors of reading difficulty, extending the use of transformer language models in psycholinguistics. It demonstrates that the energy measure from these models robustly predicts reading times across multiple corpora, outperforming traditional metrics like surprisal and attention entropy. In a controlled experiment on relative clause processing, energy captures known asymmetries, suggesting it may unify previously complementary predictors.
arXiv AI
Jun 2
Generic Interpretation Approach for Transformer Models Incorporating Heterogenous Attention Structures
arXiv:2605. 27458v2 Announce Type: replace-cross Abstract: Transformer has significantly propelled the development of artificial intelligence, and certainly the development of agents as well.
arXiv Machine Learning
Oct 19
Anisotropic Representation Dynamics in Transformers
The paper introduces a Mixture‑Gated Theory of Anisotropic Representation Dynamics that decomposes Transformer self‑attention into value transport, query gating, and key gating. It shows that the geometry of attended representations can cause a representation difference to contract or expand, a phenomenon termed gating‑induced regime switching. The authors derive a spectral criterion to predict switching, quantify its strength, and identify subspaces where switching occurs, validating these predictions experimentally across four Transformer architectures.
arXiv Machine Learning
Oct 22
Kinetic theory for Transformers and the lost-in-the-middle phenomenon
The paper investigates causal self‑attention dynamics in decoder Transformers, modeling them as a non‑exchangeable interacting particle system. Using cumulant expansions and Glauber calculus, the authors prove a quantitative mean‑field limit and a next‑order correlation characterization. For i.i.d. uniformly distributed tokens, they solve the limiting correlation equation in closed form and rigorously explain the empirically observed lost‑in‑the‑middle phenomenon, showing a U‑shaped token retrieval profile with primacy, recency, and a unique interior minimum under a smallness condition.