arXiv Machine Learning
1d ago

Group-Invariant Statistics Determine Embedding Geometry: Harmonic Analysis of Representations from Bach to the Night Sky

The paper shows that the geometric patterns seen in language model embeddings—such as circles for months and saddle-shaped manifolds—arise from group-invariant statistics in word co‑occurrence data. By extending previous work on translation symmetry to arbitrary finite, compact, and homogeneous groups, the authors prove that embeddings correspond to matrix elements of the irreducible representations of the symmetry group. They validate this theory experimentally with the cyclic group <Z_{12}> for months, a dihedral group for musical chords, and a spherical‑harmonic embedding for celestial objects.

By Liam Storan, Andreas Tolias, Nina Miolane