arXiv Machine Learning
Multi-Head Self Attention is a Parameter Identification Mechanism
Read the original on arXiv Machine Learning →The Flow has not summarised this story yet — read it at arXiv Machine Learning.
Related stories
arXiv Machine Learning
Sep 2
Performance-Efficiency Tradeoffs in Transformers: An Approximation Theory Perspective
The paper examines how to allocate attention heads and head dimensions across Transformer layers to balance expressivity and efficiency. It provides a mathematical analysis of early layers’ role in information extraction and characterizes the trade‑off between head count and dimension under a fixed parameter budget. The authors prove a saturation effect of softmax activations, showing that increasing head dimensions yields diminishing returns, especially for long sequences, and propose strategies for efficient parameter allocation across layers.
arXiv Machine Learning
Jun 29
Prism Transformer: Progressive Head Schedules for Hierarchical Attention Processing
arXiv:2606. 27449v1 Announce Type: new Abstract: Multi-head attention conventionally partitions the hidden dimension equally across all heads at every layer, enforcing an identical representational subspace dimension (dh = dmodel/h) throughout the models depth.
arXiv Machine Learning
Oct 15
Choosing the Number of Attention Heads from Intrinsic Dimension: A Small-Scale Study
arXiv:2610.19966v1 Announce Type: cross Abstract: The number of attention heads in a transformer is usually set by convention or grid search. We test a simple heuristic for setting it in advance: est...
arXiv Machine Learning
Oct 12
Data-Adaptive Mahalanobis Metric Learning for Cross-Head Attention in Transformers
The paper introduces Data-Adaptive Mahalanobis Cross-Head Attention (DAMCHA), a new attention mechanism that extends Multi-Head Attention by learning a full cross-head metric matrix and adapting it to input data. DAMCHA captures inter-head interactions beyond the block-diagonal structure of standard MHA and uses input-dependent parameterization for dynamic attention geometry. The authors provide regularization techniques, such as stack-wise parameter sharing, to maintain efficiency and stability, and report that DAMCHA-based Transformers outperform baselines on several benchmarks.
arXiv Machine Learning
Jun 5
Specialization of softmax attention heads: insights from the high-dimensional single-location model
arXiv:2603. 03993v2 Announce Type: replace Abstract: Multi-head attention enables transformer models to represent multiple attention patterns simultaneously.
arXiv Machine Learning
Jul 22
Relative Positions Generalize, Absolute Positions Memorize: An Implicit-Bias Account of Length Generalization in Attention
arXiv:2607. 18759v1 Announce Type: new Abstract: Transformers with relative positional encodings often extrapolate to sequences longer than those seen during training, whereas transformers with learned absolute encodings typically do not.