Selective state space models (SSMs) use a recurrence to mix token information, a process analogous to attention in transformers. By modeling token evolution as an ordinary differential equation and applying input‑to‑state stability, the study proves that SSMs exhibit local exponential stability of consensus equilibria and delineates their domain of attraction for time‑varying weight matrices. Experiments on a pretrained Mamba‑2 model reveal that the output gate controls the degree of consensus, preventing tokens from fully converging.
By Jo\~ao Pedro Silvestre, \'Alvaro Rodr\'iguez Abella, Paulo Tabuada
arXiv:2608. 08922v1 Announce Type: cross Abstract: Transformer layers generate state-dependent interaction networks: token representations determine the attention matrix, which in turn updates the representations.
By Qucheng Gao, Zuyi Yang, Xiao Chen
arXiv:2501. 18322v2 Announce Type: replace Abstract: Transformers, which are state-of-the-art in most machine learning tasks, represent the data as sequences of vectors called tokens.
By Val\'erie Castin, Pierre Ablin, Jos\'e Antonio Carrillo, Gabriel Peyr\'e
arXiv:2609.37921v1 Announce Type: new
Abstract: What are the inductive biases of a Transformer architecture? Existing theory on how the forward pass shapes representations either considers whether Tr...
By Erkan Turan, Gaspard Abel, Maks Ovsjanikov
arXiv:2606. 16730v2 Announce Type: replace-cross Abstract: We re-interpret Transformer pretraining as a fast-slow, singularly perturbed flow along depth, with untied weights as its non-autonomous feature.
By Zhengyuan Gao
arXiv:2602. 19143v2 Announce Type: replace Abstract: This paper studies simple transformers trained on a high-order Markov chain, where the model must incorporate information from multiple past positions, each with different statistical importance.
By O\u{g}uz Kaan Y\"uksel, Rodrigo Alvarez Lucendo, Nicolas Flammarion
We study the inference-time behavior of deep linear encoder-only transformers through the lens of interacting particle systems. In this perspective, tokens are modeled as particles that interact dynamically through successive linear self-attention layers.
arXiv:2410. 24050v3 Announce Type: replace Abstract: Large-scale pretraining of transformers has been central to the success of foundation models.
By Ambroise Odonnat, Wassim Bouaziz, Vivien Cabannes
arXiv:2609.28448v1 Announce Type: cross
Abstract: We study the nonequilibrium dynamics of a minimal recurrent transformer with $N$ normalized tokens, $Q=K=I$, and a negative value map $V=-I$. Similar...
By Qucheng Gao, Zuyi Yang, Xiao Chen
arXiv:2607. 18584v1 Announce Type: new Abstract: We study the inference-time behavior of deep linear encoder-only transformers through the lens of interacting particle systems.
By Sixu Li, Thomas Jacob Maranzatto, Jan Peszek, Trevor Teolis, Semih Akkoc, Konstantin Riedl, Sennur Ulukus, Nicol\'as Garc\'ia Trillos
arXiv:2603. 06591v2 Announce Type: replace Abstract: Transformers frequently allocate disproportionate attention to specific tokens, a phenomenon known as attention sinks.
By Runyu Peng, Ruixiao Li, Mingshu Chen, Yunhua Zhou, Qipeng Guo, Xipeng Qiu, Yucheng Lu, Chen Zhao
arXiv:2510. 09904v2 Announce Type: replace-cross Abstract: Despite their widespread use, training deep Transformers can be unstable.
By Kelvin Kan, Xingjian Li, Benjamin J. Zhang, Tuhin Sahai, Stanley Osher, Krishna Kumar, Markos A. Katsoulakis