The paper investigates causal self‑attention dynamics in decoder Transformers, modeling them as a non‑exchangeable interacting particle system. Using cumulant expansions and Glauber calculus, the authors prove a quantitative mean‑field limit and a next‑order correlation characterization. For i.i.d. uniformly distributed tokens, they solve the limiting correlation equation in closed form and rigorously explain the empirically observed lost‑in‑the‑middle phenomenon, showing a U‑shaped token retrieval profile with primacy, recency, and a unique interior minimum under a smallness condition.
By Mitia Duerinckx, Borjan Geshkovski, Stefano Rossi
The paper introduces SympFormer, a framework that derives accelerated attention blocks from inertial, Nesterov-type damped Hamiltonian dynamics on spaces of probability densities. It develops new geometries—local-mobility for Otto-Wasserstein and softmax-Wasserstein, and a quotient Stein geometry—to rigorously formulate these dynamics, providing gradient-flow and Hamiltonian foundations for softmax and linear attention. The authors analyze existence conditions, link continuous moment blow-up to discrete layer depth, and propose geometric corrections that empirically improve next-token prediction in decoder-only models.
By Viktor Stein, Levin Maier, Wuchen Li, Gabriele Steidl
arXiv:2501. 18322v2 Announce Type: replace Abstract: Transformers, which are state-of-the-art in most machine learning tasks, represent the data as sequences of vectors called tokens.
By Val\'erie Castin, Pierre Ablin, Jos\'e Antonio Carrillo, Gabriel Peyr\'e
arXiv:2607. 18584v1 Announce Type: new Abstract: We study the inference-time behavior of deep linear encoder-only transformers through the lens of interacting particle systems.
By Sixu Li, Thomas Jacob Maranzatto, Jan Peszek, Trevor Teolis, Semih Akkoc, Konstantin Riedl, Sennur Ulukus, Nicol\'as Garc\'ia Trillos
arXiv:2606. 07600v1 Announce Type: cross Abstract: We formulate data propagation through the Transformer, the machine learning architecture powering large language models, as a nonlinear control system on the space of probability measures.
By Albert Alcalde, Zhengping Ji, Enrique Zuazua
arXiv:2512. 10656v3 Announce Type: replace Abstract: As context windows in large language models continue to expand, it is essential to characterize how attention behaves at extreme sequence lengths.
By L\'ea Bohbot, Cyril Letrouit, Gabriel Peyr\'e, Fran\c{c}ois-Xavier Vialard
arXiv:2409. 08469v4 Announce Type: replace-cross Abstract: We provide finite-particle convergence rates for the Stein Variational Gradient Descent (SVGD) algorithm in the Kernelized Stein Discrepancy ($\mathsf{KSD}$) and Wasserstein-2 metrics.
By Sayan Banerjee, Krishnakumar Balasubramanian, Promit Ghosal
arXiv:2610. 19773v1 Announce Type: new Abstract: Score-based generative models driven by $\alpha$-stable L{\'e}vy processes have been proposed as an alternative to Brownian diffusion models, with empirical evidence suggesting improved sampling and better modeling of heavy-tailed data.
By Ianis Hammani (ENS Paris Saclay, LMO), Claire Boyer (LMO, IUF)
Selective state space models (SSMs) use a recurrence to mix token information, a process analogous to attention in transformers. By modeling token evolution as an ordinary differential equation and applying input‑to‑state stability, the study proves that SSMs exhibit local exponential stability of consensus equilibria and delineates their domain of attraction for time‑varying weight matrices. Experiments on a pretrained Mamba‑2 model reveal that the output gate controls the degree of consensus, preventing tokens from fully converging.
By Jo\~ao Pedro Silvestre, \'Alvaro Rodr\'iguez Abella, Paulo Tabuada
arXiv:2602. 18849v2 Announce Type: replace-cross Abstract: We develop a sensitivity analysis for transformer attention in a geometry aligned with tokenwise computation.
By Seyed Morteza Emadi
arXiv:2605.07772v2 Announce Type: replace
Abstract: Transformers perform inference by iteratively transforming token representations across layers. This layerwise computation has been studied empiric...
By Noboru Isobe, Daisuke Inoue, Masaaki Imaizumi
arXiv:2609. 11837v1 Announce Type: cross Abstract: We study the nonlocal continuity equation \[ \partial_t\mu_b =\operatorname{div}\!
By Andrea Agazzi, Giuseppe Bruno, Federico Pasqualotto, Philippe Rigollet