arXiv AI
Jun 8
Limitations of Normalization in Attention Mechanism
arXiv:2508. 17821v3 Announce Type: replace-cross Abstract: This paper investigates the limitations of the normalization in attention mechanisms.
Specialization of softmax attention heads: insights from the high-dimensional single-location model
arXiv:2603. 03993v2 Announce Type: replace Abstract: Multi-head attention enables transformer models to represent multiple attention patterns simultaneously.
Related stories
arXiv Machine Learning
Jun 11
Phase Transitions in Attention: A Bayesian Theory of Copy Head Emergence
arXiv:2606. 12058v1 Announce Type: cross Abstract: Attention is the key mechanism underlying in-context learning in transformers, and attention patterns have been observed empirically to emerge abruptly during training.
arXiv Machine Learning
Aug 10
Pay Attention to Attention Distribution: A New Local Lipschitz Bound for Transformers
arXiv:2507. 07814v2 Announce Type: replace Abstract: We introduce a novel upper bound on the local Lipschitz constant of the dot-product self-attention block showing its dependence on the attention map distributions.
arXiv Machine Learning
Jun 17
Softmax as Linear Attention in the Large-Prompt Regime: a Measure-based Perspective
arXiv:2512. 11784v2 Announce Type: replace Abstract: Softmax attention is a central component of transformer architectures, yet its nonlinear structure poses significant challenges for theoretical analysis.
arXiv Machine Learning
Jul 22
Relative Positions Generalize, Absolute Positions Memorize: An Implicit-Bias Account of Length Generalization in Attention
arXiv:2607. 18759v1 Announce Type: new Abstract: Transformers with relative positional encodings often extrapolate to sequences longer than those seen during training, whereas transformers with learned absolute encodings typically do not.
arXiv Machine Learning
Jul 2
Ghost in the Kernel: In-Context Learning with Efficient Transformers via Domain Generalization
arXiv:2607. 00479v1 Announce Type: new Abstract: Transformer-based large models have demonstrated remarkable generalization abilities across different tasks by leveraging a context-aware attention module for in-context learning.
arXiv Machine Learning
Jun 24
Asymptotic Signal Subspace Recovery in Softmax Attention Models
arXiv:2606. 22406v2 Announce Type: replace Abstract: Attention mechanisms have demonstrated remarkable empirical success in identifying relevant information from large collections of tokens, yet the theoretical principles underlying this behavior remain poorly understood.
arXiv Machine Learning
Jun 5
High-Dimensional Theory of LoRA Fine-Tuning in a Solvable Attention Model
arXiv:2606. 05899v1 Announce Type: new Abstract: We develop a high-dimensional statistical theory of low-rank adaptation (LoRA) in attention models, capturing the interplay between pre-training and fine-tuning.
arXiv Machine Learning
Jul 7
Incremental Learning of Sparse Attention Patterns in Transformers
arXiv:2602. 19143v2 Announce Type: replace Abstract: This paper studies simple transformers trained on a high-order Markov chain, where the model must incorporate information from multiple past positions, each with different statistical importance.
arXiv Machine Learning
Aug 11
Training-Free Universal Approximation by Prompting Random Transformers
arXiv:2608. 09558v1 Announce Type: new Abstract: How expressive is prompting a transformer?
arXiv AI
Jun 29
Flexformer: Flexible Linear Transformer with Learnable Attention Kernel
arXiv:2606. 27748v1 Announce Type: cross Abstract: Transformer models rely on attention mechanism to capture long-range dependencies but suffer from quadratic complexity, limiting their scalability to long sequences.
arXiv Machine Learning
Jun 4
Customizing the Inductive Biases of Softmax Attention using Structured Matrices
arXiv:2509. 07963v2 Announce Type: replace Abstract: The core component of attention is the scoring function, which transforms the inputs into low-dimensional queries and keys and takes the dot product of each pair.