arXiv:2609.15975v1 Announce Type: cross
Abstract: Transformer representations evolve through learned additive transformations that either preserve their current direction or redirect it. We study thi...
By Shwai He, Haichao Zhang, Shen Yan
arXiv:2501. 18322v2 Announce Type: replace Abstract: Transformers, which are state-of-the-art in most machine learning tasks, represent the data as sequences of vectors called tokens.
By Val\'erie Castin, Pierre Ablin, Jos\'e Antonio Carrillo, Gabriel Peyr\'e
arXiv:2606. 25010v1 Announce Type: new Abstract: Neural scaling laws for transformer language models predict smooth improvements in pretraining loss with increasing parameters, but downstream capabilities such as in-context learning are known to emerge abruptly past a certain model scale.
By Vatsal Baherwani, Zixi Chen, Shikai Qiu, Andrew Gordon Wilson, Pavel Izmailov
arXiv:2410. 24050v3 Announce Type: replace Abstract: Large-scale pretraining of transformers has been central to the success of foundation models.
By Ambroise Odonnat, Wassim Bouaziz, Vivien Cabannes
arXiv:2602. 19143v2 Announce Type: replace Abstract: This paper studies simple transformers trained on a high-order Markov chain, where the model must incorporate information from multiple past positions, each with different statistical importance.
By O\u{g}uz Kaan Y\"uksel, Rodrigo Alvarez Lucendo, Nicolas Flammarion
The paper introduces a Mixture‑Gated Theory of Anisotropic Representation Dynamics that decomposes Transformer self‑attention into value transport, query gating, and key gating. It shows that the geometry of attended representations can cause a representation difference to contract or expand, a phenomenon termed gating‑induced regime switching. The authors derive a spectral criterion to predict switching, quantify its strength, and identify subspaces where switching occurs, validating these predictions experimentally across four Transformer architectures.
By Nguyen Do, My T. Thai
arXiv:2605.07772v2 Announce Type: replace
Abstract: Transformers perform inference by iteratively transforming token representations across layers. This layerwise computation has been studied empiric...
By Noboru Isobe, Daisuke Inoue, Masaaki Imaizumi
arXiv:2607. 11875v1 Announce Type: cross Abstract: We present a theoretical framework to explain the emergence of inductive reasoning abilities in Transformer language models.
By Tiberiu Musat, Tiago Pimentel, Nicholas Zucchet, Thomas Hofmann
We present a theoretical framework to explain the emergence of inductive reasoning abilities in Transformer language models. While previous works on Transformer learning dynamics have so far been mostly tied to specific tasks, we study a generalized class of inductive tasks that unifies several synthetic tasks known in the literature, including in-context n-grams and multi-hop reasoning.
arXiv:2606. 01306v1 Announce Type: new Abstract: While Transformer-based architectures have established themselves as a dominant paradigm in Multivariate Time Series Forecasting (MTSF), their core self-attention mechanism inherently functions as a low-pass filter, systematically smoothing out high-frequency signals vital for sharp local changes.
By Peng He, Yao Liu, Yanglei Gan, Run Lin, Yuxiang Cai, Qiao Liu
arXiv:2606. 17830v1 Announce Type: cross Abstract: Neural network parameter spaces are inherently non-injective, as distinct parameter configurations can realize identical functions through functional equivalence.
By Viet-Hoang Tran, Vinh Khanh Bui, Van-Hoan Trinh, Tan Lai Ngoc, Tan M. Nguyen
arXiv:2610.23242v2 Announce Type: replace
Abstract: Small differences between internal representations can shrink or grow as they pass through a Transformer layer, affecting how information is preser...
By Nguyen Do, My T. Thai