← Back to all news
arXiv Computation and Language By Samuele Vallisa, Federico Ravenda, Claudio Palominos, Rui He, Andrea Raballo, Antonietta Mira, Philipp Homan, Wolfram Hinzen

The Changing Geometry of Grammar: Dimensionality and Neighborhood Reorganization across Transformer Layers

Read the original on arXiv Computation and Language →

The paper studies how transformer representations evolve across layers by examining the intrinsic dimensionality (ID) of token embeddings and their neighborhood structures. It finds that closed‑class tokens expand and collapse earlier than open‑class tokens, and that these changes are linked to shifts in local geometry. The authors compare encoder and decoder models, showing distinct layer‑wise behaviors, and demonstrate that geometric features alone can predict a token’s part‑of‑speech and reveal how semantic content changes across layers.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computation and Language.

Related stories

Hugging Face Trending Papers
Jun 25

Structure Before Collapse: Transient semantic geometry in next-token prediction

Neural Collapse predicts that balanced one-hot classification pushes model representations to be equally far from each other; a symmetric configuration that depends only on the output label and ignores any semantic similarity in the inputs. This creates a puzzle: next-token prediction language models are trained predominantly (as context length increases) with one-hot labels: the same context is very unlikely to appear twice in training with different labels.

llmsragefficiency
More like this →
arXiv Computation and Language
Aug 31

Tracing the complexity profiles of different linguistic phenomena through the intrinsic dimension of LLM representations

The paper investigates the intrinsic dimension (ID) of large language model (LLM) representations as an indicator of linguistic complexity. By comparing ID across model layers for coordination vs. subordination, right‑branching vs. center‑embedding, and unambiguous vs. ambiguous attachment, the authors find consistent ID differences that align with established complexity contrasts. Experiments across six LLMs, including representational similarity and layer pruning analyses, confirm that more complex phenomena produce higher ID profiles, with peaks occurring at different layers for each contrast.

By Marco Baroni, Emily Cheng, Iria de-Dios-Flores, Francesca Franzon
llmsragefficiency
More like this →