arXiv Machine Learning July 1, 2026 By Dhruva Karkada, Daniel J. Korchinski, Andres Nava, Matthieu Wyart, Yasaman Bahri

Symmetry in language statistics shapes the geometry of model representations

Read the original on arXiv Machine Learning →

arXiv:2602. 15029v3 Announce Type: replace Abstract: The internal representations learned by language models consistently exhibit striking geometric structure: calendar months organize into a circle, historical years form a smooth one-dimensional manifold, and cities' latitudes and longitudes can be decoded using a linear probe.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

arXiv Machine Learning

1d ago

Group-Invariant Statistics Determine Embedding Geometry: Harmonic Analysis of Representations from Bach to the Night Sky

The paper shows that the geometric patterns seen in language model embeddings—such as circles for months and saddle-shaped manifolds—arise from group-invariant statistics in word co‑occurrence data. By extending previous work on translation symmetry to arbitrary finite, compact, and homogeneous groups, the authors prove that embeddings correspond to matrix elements of the irreducible representations of the symmetry group. They validate this theory experimentally with the cyclic group <Z_{12}> for months, a dihedral group for musical chords, and a spherical‑harmonic embedding for celestial objects.

By Liam Storan, Andreas Tolias, Nina Miolane

llms rag

Hugging Face Trending Papers

Jul 5

Language Models Represent and Transform Concepts with Shared Geometry

How concepts are represented in neural networks is a fundamental question in machine learning. The dominant view treats concept representations as stationary geometric objects.

llms

arXiv AI

Jul 7

Language Models Represent and Transform Concepts with Shared Geometry

arXiv:2607. 04525v1 Announce Type: cross Abstract: How concepts are represented in neural networks is a fundamental question in machine learning.

By Zhimin Hu, Lanhao Niu, Sashank Varma

llms

arXiv AI

Sep 10

Recovering Temporal and Geographic Signals from Language Model Embeddings

arXiv:2609.05721v1 Announce Type: new Abstract: Understanding whether language-model embeddings encode structured real-world information is important for both representation analysis and information...

By Esteban Feuerstein, Victoria Klimkowski, Juan Manuel Ortiz de Zarate, Federico Hern\'an Suaiter

llms rag safety

arXiv AI

Jul 9

Riemannian Geometry for Pre-trained Language Model Embeddings

arXiv:2607. 07047v1 Announce Type: cross Abstract: Understanding the geometric structure of pre-trained language model embeddings matters for interpretability and safety.

By Szczepan Konior, Alexandre Quemy, Przemys{\l}aw Klocek, Gr\'egoire Cattan, Bart{\l}omiej Sobieski

llms rag benchmarks safety

arXiv Machine Learning

Sep 1

Context Staircase: Signature-Aligned Dynamics of Token Embeddings under Small Initialization

arXiv:2608.30315v1 Announce Type: new Abstract: Token embeddings are the basic representational units that connect discrete tokens with continuous computation in language models. Although modern lang...

By Junjie Yao, Liangkai Hang, Zhi-Qin John Xu

llms rag safety

One email a morning, machine-written

Related stories

Group-Invariant Statistics Determine Embedding Geometry: Harmonic Analysis of Representations from Bach to the Night Sky

Language Models Represent and Transform Concepts with Shared Geometry

Language Models Represent and Transform Concepts with Shared Geometry

Recovering Temporal and Geographic Signals from Language Model Embeddings

Riemannian Geometry for Pre-trained Language Model Embeddings

Context Staircase: Signature-Aligned Dynamics of Token Embeddings under Small Initialization