arXiv Machine Learning By Shang Xu

Geometric knowledge spaces, Attention pairing and Architectures

Read the original on arXiv Machine Learning →

The paper introduces a geometric framework for deriving attention mechanisms and exploring new transformer architectures. It formalizes concepts such as knowledge worlds and attention pairings, showing how this approach naturally yields standard attention, causal attention, linear attention, tensor attention, and recurrent neural networks. The authors also propose several new geometric spaces that could inspire novel neural network designs.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

arXiv Computation and Language
Aug 25

You Need Better Attention Priors

arXiv:2601.15380v2 Announce Type: replace-cross Abstract: We generalize the attention mechanism by viewing it through the lens of Entropic Optimal Transport, revealing that standard attention corresp...

By Elon Litman, Gabe Guo
arXiv Machine Learning
Oct 16

Interfaces and Modules: A Scalable Representation for Weight-Space Learning across Architectures

The paper introduces the Universal Weight Encoder (UWE), a scalable representation for weight-space learning that uses a typed graph of interfaces and modules to respect parameter symmetries across diverse neural network architectures. UWE combines local equivariant updates with global attention over invariant interface summaries, enabling a single encoder to process multiple supported architectures efficiently. Experiments show UWE matches or surpasses architecture‑specific state‑of‑the‑art methods on standard benchmarks and excels on a large zoo of 1,200 transformers, ranking unseen architectures accurately while using significantly less memory.

By Micha{\l} Kozyra, Michael Bronstein
Hugging Face Trending Papers
Oct 9

Data-Adaptive Mahalanobis Metric Learning for Cross-Head Attention in Transformers

The paper introduces Data-Adaptive Mahalanobis Cross-Head Attention (DAMCHA), a new attention mechanism that extends Multi-Head Attention by learning a full cross-head metric matrix, thereby capturing inter-head interactions beyond the block-diagonal structure of standard MHA. DAMCHA parameterizes this metric as an input-dependent function, allowing the attention geometry to adapt dynamically to the data’s intrinsic manifold structure. The authors also propose regularization techniques, such as stack-wise parameter sharing, to maintain computational efficiency and stable optimization, and demonstrate that DAMCHA-equipped Transformers outperform existing baselines on multiple benchmark tasks.