The paper investigates the training dynamics of attention mechanisms in high-dimensional settings, focusing on attention-indexed models that encompass multi-layer and multi-head architectures. It shows that while the loss landscape can be described by a finite set of trace order parameters, the online stochastic gradient descent dynamics involve an infinite hierarchy of matrix moments that can be accurately approximated by a finite truncated system. The study further reveals that the choice of attention parameterization acts as an implicit bias: untied attention can get trapped in uninformative states, whereas tied attention induces symmetry breaking and enables weak recovery with θ(d² log d) samples, and untied attention exhibits a fast-slow dynamic leading to weak recovery when symmetry is broken.
By Yizhou Xu, Margarita Sagitova, Lenka Zdeborov\'a, Florent Krzakala
arXiv:2610.08420v1 Announce Type: new
Abstract: We establish a polynomial sample complexity separation between symmetry-aware and symmetry-agnostic feature learning. We study growing-rank multi-index...
By Jivan Waber, Vanessa Piccolo, Yatin Dandi, Florent Krzakala
The paper introduces a tensor algebra, ∗_G, that enforces exact symmetry in machine‑learned models of molecular and crystal responses. By applying this algebra to the Orb‑v3 force field, the authors demonstrate that many models incorrectly alter a crystal’s space group during relaxation, and they show how projections derived from ∗_G can correct these errors without retraining. The method yields reference‑free error bounds and symmetry‑preserving low‑rank approximations, reducing space‑group changes to at most 2%.
By Paulina Hoyos, Shashanka Ubaru, Dongsung Huh, Vasileios Kalantzis, Kenneth L. Clarkson, Misha Kilmer, Haim Avron, Lior Horesh
We establish a polynomial sample complexity separation between symmetry-aware and symmetry-agnostic feature learning. We study growing-rank multi-index models with high-dimensional Gaussian covariates...
The paper investigates why diffusion models sometimes fail to learn symmetries present in data distributions, such as different orientations of a dog. Focusing on linear diffusion models, it identifies three possible sources of symmetry breaking—non‑symmetric initial weights, noisy learning targets, and finite‑sample noise—and shows that target noise is the most robust cause, persisting even with abundant data. Using random matrix theory, the authors derive formulas for average symmetry error, revealing that smaller symmetries are harder to learn.
By John J. Vastola, Kanaka Rajan
arXiv:2609.01231v1 Announce Type: new
Abstract: We prove that a multi-head scaled dot product attention can be viewed as a parameter identification strategy. The ratio of unidentified parameters to t...
By W. Ross Morrow
The paper shows that unsupervised representational alignment can fail due to symmetry in the stimulus geometry, even before data are collected. By using a design-time diagnostic based on the automorphism group of the geometry, the authors demonstrate that dense sampling can create near-duplicates that make alignment degenerate. Applying this diagnostic to a colour design reduces catastrophic alignment failures from 75% to 2% without altering models, layers, or solvers.
By Jing Xu, Christopher Kanan
arXiv:2608. 10251v1 Announce Type: cross Abstract: A transformer's answer lives on one axis: the direction its unembedding reads.
By Mark Oskin
The projection of queries and keys are central to the attention mechanism in Transformer architectures. While they are mathematically symmetric, they play different roles in attention mechanisms.
arXiv:2606. 04108v1 Announce Type: cross Abstract: Single-view 3D generative models have achieved impressive visual quality, yet they are not designed to satisfy structural or functional requirements, and in practice, often fall short.
By Guangda Ji, Qimin Chen, Qinchan Li, Mingrui Zhao, Kai Wang, Hao Zhang
arXiv:2605. 04893v2 Announce Type: replace Abstract: When a language model processes a hallucinated response, its attention routing tends to fail in one of two shapes: over-concentrating on a narrow set of positions, or spreading so diffusely that relevance is diluted, and the shape of the failure carries diagnostic signal.
By Dominik Dahlem, Diego Maniloff, Mac Misiura
arXiv:2606. 24975v1 Announce Type: new Abstract: PaTH Attention showed that replacing RoPE's position-indexed rotations with accumulated data-dependent Householder reflections yields strong length extrapolation, though performance degrades at extreme context lengths.
By Mahesh Godavarti