arXiv:2601.15380v2 Announce Type: replace-cross
Abstract: We generalize the attention mechanism by viewing it through the lens of Entropic Optimal Transport, revealing that standard attention corresp...
By Elon Litman, Gabe Guo
The paper introduces the Universal Weight Encoder (UWE), a scalable representation for weight-space learning that uses a typed graph of interfaces and modules to respect parameter symmetries across diverse neural network architectures. UWE combines local equivariant updates with global attention over invariant interface summaries, enabling a single encoder to process multiple supported architectures efficiently. Experiments show UWE matches or surpasses architecture‑specific state‑of‑the‑art methods on standard benchmarks and excels on a large zoo of 1,200 transformers, ranking unseen architectures accurately while using significantly less memory.
By Micha{\l} Kozyra, Michael Bronstein
arXiv:2512. 12225v3 Announce Type: replace Abstract: Developing artificial agents that unify representation, memory, adaptation, and prediction remains a fundamental challenge in artificial intelligence.
By Laha Ale
The paper introduces Data-Adaptive Mahalanobis Cross-Head Attention (DAMCHA), a new attention mechanism that extends Multi-Head Attention by learning a full cross-head metric matrix, thereby capturing inter-head interactions beyond the block-diagonal structure of standard MHA. DAMCHA parameterizes this metric as an input-dependent function, allowing the attention geometry to adapt dynamically to the data’s intrinsic manifold structure. The authors also propose regularization techniques, such as stack-wise parameter sharing, to maintain computational efficiency and stable optimization, and demonstrate that DAMCHA-equipped Transformers outperform existing baselines on multiple benchmark tasks.
Positional encodings (PEs) are essential for Transformers. Yet designing effective PEs for non-Euclidean graphs remains challenging.
arXiv:2601. 22402v2 Announce Type: replace-cross Abstract: Rotary Positional Embeddings (RoPE) have become the standard for Large Language Models (LLMs) due to their ability to encode relative positions through geometric rotation.
By Kanishk Awadhiya