arXiv:2501. 18322v2 Announce Type: replace Abstract: Transformers, which are state-of-the-art in most machine learning tasks, represent the data as sequences of vectors called tokens.
arXiv:2606. 12058v1 Announce Type: cross Abstract: Attention is the key mechanism underlying in-context learning in transformers, and attention patterns have been observed empirically to emerge abruptly during training.
The paper investigates energy-based transformers as predictors of reading difficulty, extending the use of transformer language models in psycholinguistics. It demonstrates that the energy measure from these models robustly predicts reading times across multiple corpora, outperforming traditional metrics like surprisal and attention entropy. In a controlled experiment on relative clause processing, energy captures known asymmetries, suggesting it may unify previously complementary predictors.
arXiv:2605. 27458v2 Announce Type: replace-cross Abstract: Transformer has significantly propelled the development of artificial intelligence, and certainly the development of agents as well.
The paper introduces a Mixture‑Gated Theory of Anisotropic Representation Dynamics that decomposes Transformer self‑attention into value transport, query gating, and key gating. It shows that the geometry of attended representations can cause a representation difference to contract or expand, a phenomenon termed gating‑induced regime switching. The authors derive a spectral criterion to predict switching, quantify its strength, and identify subspaces where switching occurs, validating these predictions experimentally across four Transformer architectures.
The paper investigates causal self‑attention dynamics in decoder Transformers, modeling them as a non‑exchangeable interacting particle system. Using cumulant expansions and Glauber calculus, the authors prove a quantitative mean‑field limit and a next‑order correlation characterization. For i.i.d. uniformly distributed tokens, they solve the limiting correlation equation in closed form and rigorously explain the empirically observed lost‑in‑the‑middle phenomenon, showing a U‑shaped token retrieval profile with primacy, recency, and a unique interior minimum under a smallness condition.
arXiv:2602. 19143v2 Announce Type: replace Abstract: This paper studies simple transformers trained on a high-order Markov chain, where the model must incorporate information from multiple past positions, each with different statistical importance.
arXiv:2609.07086v1 Announce Type: new Abstract: Transformers are a dominant architecture in modern machine learning, powering applications across vision, language, and beyond. At the core of their su...
arXiv:2610.23242v2 Announce Type: replace Abstract: Small differences between internal representations can shrink or grow as they pass through a Transformer layer, affecting how information is preser...
arXiv:2607. 15449v1 Announce Type: new Abstract: Using the language of Wilsonian renormalization group theory (RG), we treat the Transformer's attention mechanism as a perturbation of the trained MLP residual-stack fixed point and ask whether it constitutes a relevant, marginal, or irrelevant operator.
arXiv:2605.07772v2 Announce Type: replace Abstract: Transformers perform inference by iteratively transforming token representations across layers. This layerwise computation has been studied empiric...
The paper demonstrates that the transformer architecture naturally arises as the score of last‑token laws in a statistical model of language where sentences are treated as empirical measures of tokens. By applying the maximum‑entropy principle to a prior that assumes no inherent sentence structure, the authors derive a statistic capturing token interactions via Luce’s choice axiom, leading to the softmax choice rule and attention mechanisms. The resulting framework explains key transformer components—query‑key and value matrices as Lagrange multipliers, causal masking, multi‑head attention, and residual blocks—as consequences of entropic optimization.