arXiv Machine Learning
Aug 4
Feed-Forward Steering in Transformer Residual Dynamics
arXiv:2608. 02071v1 Announce Type: new Abstract: Attention-only dynamical theories model Transformer residual directions as particles aggregating on a sphere.
The Transformer as a Polar State Estimator
arXiv:2605. 11007v2 Announce Type: replace-cross Abstract: We show that the core components of the Transformer -- attention, residual connections, and normalization -- arise naturally from a single geometric state estimation problem.
Related stories
arXiv AI
Jun 30
Transformer Architectures as Complete Bayes Processes: A Formal Proof in the Measure-Theoretic Kernel Framework
arXiv:2606. 30440v1 Announce Type: cross Abstract: We present a complete formal proof that transformer architectures, when their internal update mechanisms satisfy a Bayes joint-distribution condition, implement exact Bayesian posterior inference.
arXiv AI
Jul 28
A Coulomb Particle Model for Learning Kernel Attention in Transformers
arXiv:2607. 23869v1 Announce Type: cross Abstract: Randomized features provide a scalable approximation to kernel machines, but their performance depends strongly on the choice of feature distribution.
arXiv AI
Jul 20
In-context learning of closed form solution to simple linear regression task using transformer with linear self-attention
arXiv:2607. 15819v1 Announce Type: cross Abstract: In-context learning is a remarkable property of transformers and has recently received a lot of interest.
arXiv AI
Jun 2
Generic Interpretation Approach for Transformer Models Incorporating Heterogenous Attention Structures
arXiv:2605. 27458v2 Announce Type: replace-cross Abstract: Transformer has significantly propelled the development of artificial intelligence, and certainly the development of agents as well.
arXiv Machine Learning
Jun 5
SpanNorm: Reconciling Training Stability and Performance in Deep Transformers
arXiv:2601. 22580v2 Announce Type: replace-cross Abstract: The success of Large Language Models (LLMs) hinges on the stable training of deep Transformer architectures.
arXiv Machine Learning
Jun 4
Vision Transformer Finetuning Benefits from Non-Smooth Components
arXiv:2602. 06883v3 Announce Type: replace Abstract: The smoothness of the transformer architecture has been extensively studied in the context of generalization, training stability, and adversarial robustness.
arXiv AI
Jun 17
Functional Equivalence in Attention: A Comprehensive Study with Applications to Linear Mode Connectivity
arXiv:2606. 17830v1 Announce Type: cross Abstract: Neural network parameter spaces are inherently non-injective, as distinct parameter configurations can realize identical functions through functional equivalence.
arXiv AI
Jun 16
Adaptive inference and function vectors in deep transformers
arXiv:2606. 16694v1 Announce Type: cross Abstract: Transformers are widely used as a general-purpose substrate for learning complex correlations between a large collection of coupled variables, but their internal mechanisms have remained mysterious.
arXiv Machine Learning
Jul 22
Toward Manifest Relationality in Transformers via Symmetry Reduction
arXiv:2602. 18948v2 Announce Type: replace Abstract: Transformer models contain substantial internal redundancy arising from coordinate-dependent representations and continuous symmetries, in model space and in head space, respectively.
arXiv AI
1d ago
Gradient Heterogeneity Complements Hessian Heterogeneity in Transformer Optimization
arXiv:2502. 00213v5 Announce Type: replace-cross Abstract: Transformers are difficult to optimize with stochastic gradient descent (SGD) and largely rely on adaptive optimizers such as Adam.
arXiv AI
Jul 10
The Phasor Transformer: Resolving Attention Bottlenecks on the Unit Circle
arXiv:2603. 17433v2 Announce Type: replace-cross Abstract: Transformer models have redefined sequence learning, yet dot-product self-attention introduces a quadratic token-mixing bottleneck for long-context time-series.