arXiv Machine Learning
Jun 30
A Mechanistic Study of Transformers Training Dynamics
arXiv:2410. 24050v3 Announce Type: replace Abstract: Large-scale pretraining of transformers has been central to the success of foundation models.
Transformer Circuits Can Realize Clustering Algorithms
arXiv:2506. 19125v2 Announce Type: replace-cross Abstract: Although transformers are most commonly optimized as statistical sequence models, it is unclear to what extent they can implement and learn exact algorithmic computations.
Related stories
arXiv Machine Learning
Jul 31
A novel k-means clustering approach using two distance measures for Gaussian data
arXiv:2511. 17823v2 Announce Type: replace Abstract: Clustering algorithms have long been the topic of research, representing the more popular side of unsupervised learning.
arXiv Machine Learning
Jun 2
Looped Transformers with Layer Normalization Provably Learn the Power Method
arXiv:2606. 00605v1 Announce Type: new Abstract: Transformers have achieved remarkable success across a wide range of applications, and a growing body of work suggests that part of their strength comes from their ability to learn and execute algorithmic procedures.
arXiv Machine Learning
Jun 19
A Unified Perspective on the Dynamics of Deep Transformers
arXiv:2501. 18322v2 Announce Type: replace Abstract: Transformers, which are state-of-the-art in most machine learning tasks, represent the data as sequences of vectors called tokens.
arXiv AI
Jul 9
Multiplication Beyond Groups: Stratified Fourier Mechanisms in Transformer Circuits
arXiv:2607. 07066v1 Announce Type: cross Abstract: Transformers have demonstrated a remarkable ability to learn algorithmic reasoning, yet mechanistic analyses have mostly focused on globally invertible operations such as cyclic addition and group composition.
arXiv AI
Jun 16
Beyond Self-Attention: Sub-Quadratic Vision Transformers for Fast Image Captioning
arXiv:2606. 14753v1 Announce Type: cross Abstract: Image captioning is a challenging and significant task that aims to generate coherent and semantically meaningful textual descriptions for given images.
arXiv Machine Learning
Jun 5
SpanNorm: Reconciling Training Stability and Performance in Deep Transformers
arXiv:2601. 22580v2 Announce Type: replace-cross Abstract: The success of Large Language Models (LLMs) hinges on the stable training of deep Transformer architectures.
arXiv AI
Jul 2
Post-Training Pruning for Diffusion Transformers
arXiv:2607. 00927v1 Announce Type: cross Abstract: Diffusion Transformers (DiTs) have demonstrated impressive performance in image generation but suffer from substantial computational overhead and resource consumption.
arXiv Machine Learning
Jul 7
Sensitivity Sampling with Predictions for k-Means Clustering
arXiv:2607. 04949v1 Announce Type: new Abstract: We study the problem of k-means clustering on large datasets.
arXiv AI
Jul 8
Transformers converge to invariant algorithmic cores
arXiv:2602. 22600v2 Announce Type: replace-cross Abstract: Training selects for behavior, not circuitry: many weight configurations can implement the same function.
arXiv AI
Jul 14
Invariant Learning Dynamics of Transformers in Inductive Reasoning Tasks
arXiv:2607. 11875v1 Announce Type: cross Abstract: We present a theoretical framework to explain the emergence of inductive reasoning abilities in Transformer language models.
arXiv AI
23h ago
Gradient Heterogeneity Complements Hessian Heterogeneity in Transformer Optimization
arXiv:2502. 00213v5 Announce Type: replace-cross Abstract: Transformers are difficult to optimize with stochastic gradient descent (SGD) and largely rely on adaptive optimizers such as Adam.