The paper examines how to allocate attention heads and head dimensions across Transformer layers to balance expressivity and efficiency. It provides a mathematical analysis of early layers’ role in information extraction and characterizes the trade‑off between head count and dimension under a fixed parameter budget. The authors prove a saturation effect of softmax activations, showing that increasing head dimensions yields diminishing returns, especially for long sequences, and propose strategies for efficient parameter allocation across layers.
By Ruoxi Yu, Haotian Jiang, Jingpu Cheng, Penghao Yu, Qianxiao Li, Zhong Li
arXiv:2606. 27449v1 Announce Type: new Abstract: Multi-head attention conventionally partitions the hidden dimension equally across all heads at every layer, enforcing an identical representational subspace dimension (dh = dmodel/h) throughout the models depth.
By Shubham Aggarwal
arXiv:2610.19966v1 Announce Type: cross
Abstract: The number of attention heads in a transformer is usually set by convention or grid search. We test a simple heuristic for setting it in advance: est...
By Arjun Boya, Charan Sai Bandaru, Masthanaiah Pillimetla, Kavana Chidanandappa
The paper introduces Data-Adaptive Mahalanobis Cross-Head Attention (DAMCHA), a new attention mechanism that extends Multi-Head Attention by learning a full cross-head metric matrix and adapting it to input data. DAMCHA captures inter-head interactions beyond the block-diagonal structure of standard MHA and uses input-dependent parameterization for dynamic attention geometry. The authors provide regularization techniques, such as stack-wise parameter sharing, to maintain efficiency and stability, and report that DAMCHA-based Transformers outperform baselines on several benchmarks.
By Zhenke Duan, Xin Li, Jiqun Pan, Dong Xiaofei, Hanwen Ning, Xinyuan Song
arXiv:2603. 03993v2 Announce Type: replace Abstract: Multi-head attention enables transformer models to represent multiple attention patterns simultaneously.
By M. Sagitova, O. Duranthon, L. Zdeborov\'a
arXiv:2607. 18759v1 Announce Type: new Abstract: Transformers with relative positional encodings often extrapolate to sequences longer than those seen during training, whereas transformers with learned absolute encodings typically do not.
By Subham Singh, Ashutosh Mishra, Subha Raut
arXiv:2606. 04032v1 Announce Type: cross Abstract: Transformers have become the standard solution for various AI tasks, with the query, key, and value (QKV) attention formulation playing a central role.
By Ali Kayyam, Anusha Madan Gopal, M Anthony Lewis
The paper introduces Data-Adaptive Mahalanobis Cross-Head Attention (DAMCHA), a new attention mechanism that extends Multi-Head Attention by learning a full cross-head metric matrix, thereby capturing inter-head interactions beyond the block-diagonal structure of standard MHA. DAMCHA parameterizes this metric as an input-dependent function, allowing the attention geometry to adapt dynamically to the data’s intrinsic manifold structure. The authors also propose regularization techniques, such as stack-wise parameter sharing, to maintain computational efficiency and stable optimization, and demonstrate that DAMCHA-equipped Transformers outperform existing baselines on multiple benchmark tasks.
SVD-Attention is a post‑hoc diagnostic tool that decomposes each self‑attention head’s interaction matrix into singular modes, scoring them by spectral magnitude and token‑wise activation variance on a small calibration set. The method retains only the modes that the head can amplify and that are actually activated by real data, producing an exact factorization without finetuning. Across seven pretrained models and five tasks, the analysis reveals task‑dependent redundancy patterns and shows that truncating the identified redundant layers incurs minimal performance loss while truncating other layers can cause significant degradation.
By Debopam Sanyal, Hongjie Chen, Riya Elizabeth Kuriyan, Arth Tak, Divya Kothandaraman, Akshay Mehra, Joshua Kimball, Alexey Tumanov
The paper investigates how transformer language models perform few‑shot learning for a simple addition task, showing that the ability is concentrated in a handful of attention heads. Using dimensionality reduction, the authors identify low‑dimensional subspaces—three heads with six‑dimensional spaces in Llama‑3‑8B‑Instruct—where specific dimensions encode the units digit via trigonometric patterns and magnitude via low‑frequency components. They also derive a mathematical identity linking aggregator and extractor subspaces, enabling tracking of information flow from examples to the final prediction.
By Xinyan Hu, Kayo Yin, Michael I. Jordan, Jacob Steinhardt, Lijie Chen
arXiv:2609.36448v1 Announce Type: new
Abstract: Transformers have demonstrated remarkable in-context learning (ICL) capabilities, enabling them to perform new tasks without additional fine-tuning. Ho...
By Junze Deng, Daouda Sow, Sen Lin, Yingbin Liang
arXiv:2608. 09558v1 Announce Type: new Abstract: How expressive is prompting a transformer?
By Alexander Hsu, Rongjie Lai