arXiv:2603. 06187v2 Announce Type: replace-cross Abstract: We introduce the Random Quadratic Form (RQF): a stochastic differential equation which formally corresponds to the gradient flow of a random quadratic functional on a sphere.
arXiv:2608. 16664v1 Announce Type: cross Abstract: We study the Random Quadratic Form (RQF) on a sphere in the presence of random Brownian forcing.
The paper investigates the discrete-time self‑attention dynamics of transformers, providing a full classification of equilibria and their stability for both single‑head and multi‑head cases. It shows that the main attractor is a consensus equilibrium aligned with the principal eigenvector of the value matrix, while locally asymptotically stable bipartite equilibria also exist in the single‑head scenario. The theoretical findings are corroborated by examining learned weight matrices from an actual transformer.
arXiv:2501. 18322v2 Announce Type: replace Abstract: Transformers, which are state-of-the-art in most machine learning tasks, represent the data as sequences of vectors called tokens.
arXiv:2602. 14885v2 Announce Type: replace-cross Abstract: Recurrent neural networks (RNNs) provide a theoretical framework for understanding computation in biological neural circuits, yet classical results, such as Hopfield's model of associative memory, rely on symmetric connectivity that restricts network dynamics to gradient-like flows.
arXiv:2608. 08922v1 Announce Type: cross Abstract: Transformer layers generate state-dependent interaction networks: token representations determine the attention matrix, which in turn updates the representations.
arXiv:2608. 14638v1 Announce Type: new Abstract: In this paper we study autoencoders, a special class of deep neural nets (DNNs) whose performance can be characterized via their fixed points.
The paper investigates how deep residual networks behave when their initial weights are correlated across layers. It confirms a conjecture that such correlated initializations interpolate between a Brownian stochastic differential equation (for independent weights) and an ordinary differential equation (for perfectly correlated weights). By applying a feature function to a stationary Gaussian sequence with regularly varying correlation, the authors prove that a unique critical scaling exists, leading the infinite‑depth limit to a Young differential equation driven by a Hermite process, which reduces to fractional Brownian motion when the feature function has Hermite rank one. The study shows that the correlation structure and Hermite rank of the initialization uniquely determine the critical scaling and asymptotic limit, making them meaningful hyperparameters in the asymptotic regime, whereas finite‑variance i.i.d. initialization always yields a Brownian driver regardless of distribution.
arXiv:2609.36230v1 Announce Type: new Abstract: We study the dynamical behavior of tokens in transformers from a control-theoretic perspective. Our model includes the feed-forward layer present after...
arXiv:2609.28448v1 Announce Type: cross Abstract: We study the nonequilibrium dynamics of a minimal recurrent transformer with $N$ normalized tokens, $Q=K=I$, and a negative value map $V=-I$. Similar...
The paper investigates causal self‑attention dynamics in decoder Transformers, modeling them as a non‑exchangeable interacting particle system. Using cumulant expansions and Glauber calculus, the authors prove a quantitative mean‑field limit and a next‑order correlation characterization. For i.i.d. uniformly distributed tokens, they solve the limiting correlation equation in closed form and rigorously explain the empirically observed lost‑in‑the‑middle phenomenon, showing a U‑shaped token retrieval profile with primacy, recency, and a unique interior minimum under a smallness condition.
arXiv:2609.39648v1 Announce Type: cross Abstract: Diffusion models are typically viewed as stochastic processes that transform noise into data. We take a complementary perspective: a diffusion model...
arXiv:2606. 07600v1 Announce Type: cross Abstract: We formulate data propagation through the Transformer, the machine learning architecture powering large language models, as a nonlinear control system on the space of probability measures.