arXiv Machine Learning
Scaled Idempotence in Transformer Attention: Paired OV Geometry and Shared-Value Algebras
Read the original on arXiv Machine Learning →The Flow has not summarised this story yet — read it at arXiv Machine Learning.
Related stories
arXiv Machine Learning
Oct 16
Inherent Trade-offs when Enforcing Correct Data Symmetries in Multi-head Self-attention
arXiv:2608. 30417v2 Announce Type: replace Abstract: We give a complete characterization of equivariant multi-head self-attention (MHSA): if an MHSA layer is equivariant to a symmetry group $G$, then $G$ can only act by permuting head-clusters, with QK and OV matrices satisfying an equivariance constraint tied to the group action.
arXiv AI
6d ago
Relational Composition as Parallel Transport: What Per-Entity Frames Force in Attention
The paper investigates how treating each entity in a graph or language model as having its own coordinate frame—viewed as a symmetry—constrains the form of attention mechanisms. It shows that with orthogonal transports, invariant attention scores depend only on norms and transported inner products, while equivariant messages lie in the span of the entity vectors and their transported counterparts. The authors demonstrate that standard attention corresponds to a flat shared frame, and that incorporating per‑entity frames allows relations to be represented as operators that compose via parallel transport, yielding empirical gains on synthetic relational tasks and natural language benchmarks.
arXiv Machine Learning
Aug 12
Off-Axis, On Purpose: Where a Transformer Computes Concepts and Why it Does So
arXiv:2608. 10251v1 Announce Type: cross Abstract: A transformer's answer lives on one axis: the direction its unembedding reads.
Hugging Face Trending Papers
Sep 28
Query Expansion and Key Specialization in Transformer Attention Geometry
The projection of queries and keys are central to the attention mechanism in Transformer architectures. While they are mathematically symmetric, they play different roles in attention mechanisms.
arXiv AI
Oct 15
Virtual Head Attention
Virtual Head Attention (VHA) is a lightweight attention mechanism that improves long‑context large language models by adding two linear operations—Q Premix and Linear Postmix—to recover head diversity and fuse inter‑head features without nonlinearities. VHA reduces KV cache size, training and prefill FLOPs, and maintains or surpasses model quality compared to existing methods such as GQA, MLA, and MFA across multiple scales and benchmarks. The design allows VHA to be folded into standard GQA inference pipelines, enabling efficient deployment with only two KV heads and full exploitation of Tensor Core parallelism.
arXiv Machine Learning
Sep 17
Reaching Every Position Without Searching: Rotating Sparse Wiring on the Hypercube as a Substitute for Attention
arXiv:2609. 18145v1 Announce Type: new Abstract: Attention pays, at every layer and for every input, the cost of searching for whom to connect.