arXiv AI
Jul 7
Language Models Represent and Transform Concepts with Shared Geometry
arXiv:2607. 04525v1 Announce Type: cross Abstract: How concepts are represented in neural networks is a fundamental question in machine learning.
The Changing Geometry of Grammar: Dimensionality and Neighborhood Reorganization across Transformer Layers
The paper studies how transformer representations evolve across layers by examining the intrinsic dimensionality (ID) of token embeddings and their neighborhood structures. It finds that closed‑class tokens expand and collapse earlier than open‑class tokens, and that these changes are linked to shifts in local geometry. The authors compare encoder and decoder models, showing distinct layer‑wise behaviors, and demonstrate that geometric features alone can predict a token’s part‑of‑speech and reveal how semantic content changes across layers.
Related stories
arXiv Machine Learning
Sep 1
Context Staircase: Signature-Aligned Dynamics of Token Embeddings under Small Initialization
arXiv:2608.30315v1 Announce Type: new Abstract: Token embeddings are the basic representational units that connect discrete tokens with continuous computation in language models. Although modern lang...
Hugging Face Trending Papers
Jul 5
Language Models Represent and Transform Concepts with Shared Geometry
How concepts are represented in neural networks is a fundamental question in machine learning. The dominant view treats concept representations as stationary geometric objects.
Hugging Face Trending Papers
Jun 25
Structure Before Collapse: Transient semantic geometry in next-token prediction
Neural Collapse predicts that balanced one-hot classification pushes model representations to be equally far from each other; a symmetric configuration that depends only on the output label and ignores any semantic similarity in the inputs. This creates a puzzle: next-token prediction language models are trained predominantly (as context length increases) with one-hot labels: the same context is very unlikely to appear twice in training with different labels.
arXiv Machine Learning
Jun 26
Structure Before Collapse: Transient semantic geometry in next-token prediction
arXiv:2606. 26749v1 Announce Type: new Abstract: Neural Collapse predicts that balanced one-hot classification pushes model representations to be equally far from each other; a symmetric configuration that depends only on the output label and ignores any semantic similarity in the inputs.
arXiv Computation and Language
Aug 31
Tracing the complexity profiles of different linguistic phenomena through the intrinsic dimension of LLM representations
The paper investigates the intrinsic dimension (ID) of large language model (LLM) representations as an indicator of linguistic complexity. By comparing ID across model layers for coordination vs. subordination, right‑branching vs. center‑embedding, and unambiguous vs. ambiguous attachment, the authors find consistent ID differences that align with established complexity contrasts. Experiments across six LLMs, including representational similarity and layer pruning analyses, confirm that more complex phenomena produce higher ID profiles, with peaks occurring at different layers for each contrast.
arXiv AI
Aug 11
The Transformer Revolution, Part 1: Dynamic Processing through Output-Weight Interconnections
arXiv:2608. 03921v2 Announce Type: replace Abstract: This paper offers a new interpretation of the Transformer during inference.
arXiv Computation and Language
Sep 2
Late Transformer Layers Recode Syntax Canonically: Evidence from Greek Scrambling and Cross-Layer Generalisation
arXiv:2609.00416v1 Announce Type: new Abstract: Probing studies have established that syntactic information is decodable in early and middle transformer layers, but what happens to that information i...
arXiv Machine Learning
Jul 3
Transformer Geometry Observatory TGO-II: Representational Similarity Observatory
arXiv:2607. 02386v1 Announce Type: cross Abstract: While Vision Transformers have achieved remarkable success across computer vision and language applications, the geometric evolution of their internal representations throughout training remains insufficiently understood.
arXiv AI
Jul 14
Laguerre Geometry for Interpreting Large Language Models
arXiv:2607. 10578v1 Announce Type: new Abstract: Existing hypotheses represent a concept in an LLM as a single point, a linear direction, or a Gaussian cluster, yet it remains unclear how and why such structures emerge.
arXiv AI
Aug 5
The Transformer Revolution, Part 1: Dynamic Processing through Output- Weight Interconnections
arXiv:2608. 03921v1 Announce Type: new Abstract: This paper offers a new interpretation of the Transformer during inference.
arXiv AI
5d ago
Technical Manual for a Toolkit for Measuring Contextual Individuation in Transformer Language Models
The article presents a technical manual for an open toolkit designed to measure how transformer language models individuate word meanings across different contexts. It introduces the concept of a "bridge form"—a single word that appears unchanged in multiple domains but with distinct senses—and outlines a full pipeline from specifying these forms to extracting layer-wise representations, computing silhouette-based separation metrics, and visualizing results. The manual details each design choice and its intended methodological safeguards, emphasizing that it serves as a methodological reference rather than reporting empirical findings.