The paper investigates causal self‑attention dynamics in decoder Transformers, modeling them as a non‑exchangeable interacting particle system. Using cumulant expansions and Glauber calculus, the authors prove a quantitative mean‑field limit and a next‑order correlation characterization. For i.i.d. uniformly distributed tokens, they solve the limiting correlation equation in closed form and rigorously explain the empirically observed lost‑in‑the‑middle phenomenon, showing a U‑shaped token retrieval profile with primacy, recency, and a unique interior minimum under a smallness condition.
By Mitia Duerinckx, Borjan Geshkovski, Stefano Rossi
The paper introduces SympFormer, a framework that derives accelerated attention blocks from inertial, Nesterov-type damped Hamiltonian dynamics on spaces of probability densities. It develops new geometries—local-mobility for Otto-Wasserstein and softmax-Wasserstein, and a quotient Stein geometry—to rigorously formulate these dynamics, providing gradient-flow and Hamiltonian foundations for softmax and linear attention. The authors analyze existence conditions, link continuous moment blow-up to discrete layer depth, and propose geometric corrections that empirically improve next-token prediction in decoder-only models.
By Viktor Stein, Levin Maier, Wuchen Li, Gabriele Steidl
arXiv:2501. 18322v2 Announce Type: replace Abstract: Transformers, which are state-of-the-art in most machine learning tasks, represent the data as sequences of vectors called tokens.
By Val\'erie Castin, Pierre Ablin, Jos\'e Antonio Carrillo, Gabriel Peyr\'e
arXiv:2607. 18584v1 Announce Type: new Abstract: We study the inference-time behavior of deep linear encoder-only transformers through the lens of interacting particle systems.
By Sixu Li, Thomas Jacob Maranzatto, Jan Peszek, Trevor Teolis, Semih Akkoc, Konstantin Riedl, Sennur Ulukus, Nicol\'as Garc\'ia Trillos
arXiv:2606. 07600v1 Announce Type: cross Abstract: We formulate data propagation through the Transformer, the machine learning architecture powering large language models, as a nonlinear control system on the space of probability measures.
By Albert Alcalde, Zhengping Ji, Enrique Zuazua
arXiv:2512. 10656v3 Announce Type: replace Abstract: As context windows in large language models continue to expand, it is essential to characterize how attention behaves at extreme sequence lengths.
By L\'ea Bohbot, Cyril Letrouit, Gabriel Peyr\'e, Fran\c{c}ois-Xavier Vialard