arXiv Machine Learning October 16, 2026 By T\=ikun \^Ong, Georg B\"okman

Inherent Trade-offs when Enforcing Correct Data Symmetries in Multi-head Self-attention

Read the original on arXiv Machine Learning →

arXiv:2608. 30417v2 Announce Type: replace Abstract: We give a complete characterization of equivariant multi-head self-attention (MHSA): if an MHSA layer is equivariant to a symmetry group $G$, then $G$ can only act by permuting head-clusters, with QK and OV matrices satisfying an equivariance constraint tied to the group action.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

arXiv Machine Learning

Sep 1

No Equivariant Architecture Covers All Equivariant Attention

arXiv:2608.30417v1 Announce Type: new Abstract: We give a complete characterization of equivariant multi-head self-attention (MHSA): if an MHSA layer is equivariant to a symmetry group $G$, then $G$...

By T\=ikun \^Ong

arXiv Machine Learning

Jun 29

A Unified Framework for Vision Transformers Equivariant to Discrete Subgroups of $\mathrm{O}(2)$

arXiv:2606. 27864v1 Announce Type: cross Abstract: Vision transformers have become a dominant architecture for visual recognition.

By T\={\i}kun \^Ong, Georg B\"okman

llms

arXiv Machine Learning

Sep 2

Scaled Idempotence in Transformer Attention: Paired OV Geometry and Shared-Value Algebras

arXiv:2609.01129v1 Announce Type: new Abstract: We identify a recurrent algebraic regularity in Transformer attention: a sparse subset of effective OV operators $T=OV^\top$ nearly closes under compos...

By Jiming Feng, Junliang Li

llms safety

arXiv Machine Learning

Sep 2

Multi-Head Self Attention is a Parameter Identification Mechanism

arXiv:2609.01231v1 Announce Type: new Abstract: We prove that a multi-head scaled dot product attention can be viewed as a parameter identification strategy. The ratio of unidentified parameters to t...

By W. Ross Morrow

llms safety

arXiv Machine Learning

Aug 4

Polyatomic Complexes: A topologically-informed learning representation for atomistic systems

arXiv:2409. 15600v3 Announce Type: replace Abstract: A representation of a molecule or material should be invariant to the symmetries of physics, unique, continuous, efficient and general.

By Rahul Khorana, Marcus Noack, Jin Qian

llms

arXiv AI

Jun 3

Exact equivariance, kept through training, buys zero-shot generalisation across the symmetry group

arXiv:2606. 03003v1 Announce Type: cross Abstract: A latent world model built from an equivariant encoder $E$ and an equivariant predictor $f$ inherits a provable symmetry of its training loss: when the world's dynamics genuinely carries a group $G$ acting on latents by an orthogonal representation $\rho(g)$, the one-step prediction relMSE is exactly invariant across the whole group, so fitting the dynamics on a restricted slice of orientations mathematically determines it on the entire orbit (j\v{u} y\=i f\v{a}n s\=an).

By Hongbo Wang (Stony Brook University)

One email a morning, machine-written

Related stories

No Equivariant Architecture Covers All Equivariant Attention

A Unified Framework for Vision Transformers Equivariant to Discrete Subgroups of $\mathrm{O}(2)$

Scaled Idempotence in Transformer Attention: Paired OV Geometry and Shared-Value Algebras

Multi-Head Self Attention is a Parameter Identification Mechanism

Polyatomic Complexes: A topologically-informed learning representation for atomistic systems

Exact equivariance, kept through training, buys zero-shot generalisation across the symmetry group