← Back to all news
arXiv Machine Learning By Claudio Altafini

Asymptotic Analysis of Discrete-Time Self-Attention Dynamics in Transformers

Read the original on arXiv Machine Learning →

The paper investigates the discrete-time self‑attention dynamics of transformers, providing a full classification of equilibria and their stability for both single‑head and multi‑head cases. It shows that the main attractor is a consensus equilibrium aligned with the principal eigenvector of the value matrix, while locally asymptotically stable bipartite equilibria also exist in the single‑head scenario. The theoretical findings are corroborated by examining learned weight matrices from an actual transformer.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

Related stories

arXiv AI
Sep 17

The Attention Within: Consensus Dynamics in Selective State Space Models

Selective state space models (SSMs) use a recurrence to mix token information, a process analogous to attention in transformers. By modeling token evolution as an ordinary differential equation and applying input‑to‑state stability, the study proves that SSMs exhibit local exponential stability of consensus equilibria and delineates their domain of attraction for time‑varying weight matrices. Experiments on a pretrained Mamba‑2 model reveal that the output gate controls the degree of consensus, preventing tokens from fully converging.

By Jo\~ao Pedro Silvestre, \'Alvaro Rodr\'iguez Abella, Paulo Tabuada
llms
More like this →
arXiv Machine Learning
Aug 6

A Mechanistic Analysis of Transformers for Dynamical Systems

arXiv:2512. 21113v2 Announce Type: replace Abstract: Transformers are increasingly adopted for modeling and forecasting time-series, yet their internal mechanisms remain poorly understood from a dynamical systems perspective.

By Gregory Duth\'e, Nikolaos Evangelou, Wei Liu, Ioannis G. Kevrekidis, Eleni Chatzi
llmsrag
More like this →
arXiv Machine Learning
Oct 19

Anisotropic Representation Dynamics in Transformers

The paper introduces a Mixture‑Gated Theory of Anisotropic Representation Dynamics that decomposes Transformer self‑attention into value transport, query gating, and key gating. It shows that the geometry of attended representations can cause a representation difference to contract or expand, a phenomenon termed gating‑induced regime switching. The authors derive a spectral criterion to predict switching, quantify its strength, and identify subspaces where switching occurs, validating these predictions experimentally across four Transformer architectures.

By Nguyen Do, My T. Thai
llms
More like this →
arXiv Machine Learning
Jul 7

Incremental Learning of Sparse Attention Patterns in Transformers

arXiv:2602. 19143v2 Announce Type: replace Abstract: This paper studies simple transformers trained on a high-order Markov chain, where the model must incorporate information from multiple past positions, each with different statistical importance.

By O\u{g}uz Kaan Y\"uksel, Rodrigo Alvarez Lucendo, Nicolas Flammarion
llms
More like this →