You Need Better Attention Priors
arXiv:2601.15380v2 Announce Type: replace-cross Abstract: We generalize the attention mechanism by viewing it through the lens of Entropic Optimal Transport, revealing that standard attention corresp...
The paper introduces a geometric framework for deriving attention mechanisms and exploring new transformer architectures. It formalizes concepts such as knowledge worlds and attention pairings, showing how this approach naturally yields standard attention, causal attention, linear attention, tensor attention, and recurrent neural networks. The authors also propose several new geometric spaces that could inspire novel neural network designs.
arXiv:2601.15380v2 Announce Type: replace-cross Abstract: We generalize the attention mechanism by viewing it through the lens of Entropic Optimal Transport, revealing that standard attention corresp...
The paper introduces the Universal Weight Encoder (UWE), a scalable representation for weight-space learning that uses a typed graph of interfaces and modules to respect parameter symmetries across diverse neural network architectures. UWE combines local equivariant updates with global attention over invariant interface summaries, enabling a single encoder to process multiple supported architectures efficiently. Experiments show UWE matches or surpasses architecture‑specific state‑of‑the‑art methods on standard benchmarks and excels on a large zoo of 1,200 transformers, ranking unseen architectures accurately while using significantly less memory.
arXiv:2512. 12225v3 Announce Type: replace Abstract: Developing artificial agents that unify representation, memory, adaptation, and prediction remains a fundamental challenge in artificial intelligence.
The paper introduces Data-Adaptive Mahalanobis Cross-Head Attention (DAMCHA), a new attention mechanism that extends Multi-Head Attention by learning a full cross-head metric matrix, thereby capturing inter-head interactions beyond the block-diagonal structure of standard MHA. DAMCHA parameterizes this metric as an input-dependent function, allowing the attention geometry to adapt dynamically to the data’s intrinsic manifold structure. The authors also propose regularization techniques, such as stack-wise parameter sharing, to maintain computational efficiency and stable optimization, and demonstrate that DAMCHA-equipped Transformers outperform existing baselines on multiple benchmark tasks.
Positional encodings (PEs) are essential for Transformers. Yet designing effective PEs for non-Euclidean graphs remains challenging.
arXiv:2601. 22402v2 Announce Type: replace-cross Abstract: Rotary Positional Embeddings (RoPE) have become the standard for Large Language Models (LLMs) due to their ability to encode relative positions through geometric rotation.
arXiv:2607. 28755v1 Announce Type: new Abstract: Over the last decade, neural networks have been applied to an increasingly diverse range of applications, including data with rich geometric, topological, or symmetry-related structure.
arXiv:2608. 01283v1 Announce Type: new Abstract: All Transformer-based large language models compute attention via the Euclidean inner product, an architectural choice that Dong et al.
arXiv:2606. 09951v1 Announce Type: new Abstract: During the training of large Transformer models, attention masks regulate the scope and direction of information flow across a sequence.
The paper explores whether a group of pyramidal columns in posterior temporal language regions could serve as a biological counterpart to a Transformer attention head. By mapping the functional organization of attention heads onto neurophysiological mechanisms within and across these columns, the authors propose that such a neuromorphic construction is possible, though empirical confirmation remains open. The study does not insist on reproducing the standard Transformer mathematics, allowing for alternative mechanical and mathematical instantiations consistent with biology.
arXiv:2606. 25293v1 Announce Type: new Abstract: Positional encodings (PEs) are essential for Transformers.
arXiv:2606. 17830v1 Announce Type: cross Abstract: Neural network parameter spaces are inherently non-injective, as distinct parameter configurations can realize identical functions through functional equivalence.